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Preface 


This year we organized the 20th edition of the International Conference on Smart Living 
and Public Health (COST 2023), a series that has succeeded in bringing together a 
community from different continents for 20 years and raised awareness about frail and 
dependent people’s quality of life in our societies. 

After 19 very successful conferences held in France (2003, 2009, 2017, 2022), 
Singapore (2004, 2013, 2018), Canada (2005, 2011), the UK (2006), Japan (2007), 
the USA (2008, 2014, 2019), South Korea (2010), Italy (2012), Switzerland (2015), 
China (2016), and Tunisia (2020), this 20th edition of the conference was organized by 
Yonsei University, Wonju, Republic of Korea, and was held during July 7—8, 2023. The 
theme of the conference was “Digital Health Transformation and Smart Ageing in place 
with & to disability.” 

ICOST 2023 provided a premier venue for the presentation and discussion of research 
on the design, development, deployment, and evaluation of AI for health, smart urban 
environments, assistive technologies, chronic disease management, and coaching and 
health telematics systems. ICOST 2023 aimed to understand and assess the diverse and 
disparate impact of digital technologies on public health in developing and developed 
countries. It brought together stakeholders from health care, public health, academia, and 
industry along with end users and family caregivers to explore how to utilize technologies 
to foster health, disease prevention, and independent living, and offer enhanced quality 
of life. The ICOST 2023 conference featured a dynamic program incorporating a range 
of oral and poster presentations, along with panel sessions. 

ICOST 2023 was proud to extend its hospitality to an international community 
consisting of researchers from major universities and research centers, representatives 
from industry, and users from 17 different countries. We would like to thank the authors 
for submitting their current research work and the Program Committee members for 
their commitment to 41 submissions were received and were sent for review. All papers 
have undergo a double-blind review process with an average of 3 reviewers per paper. 
18 papers have been accepted as full papers with an acceptance rate of around 44%. The 
ICOST proceedings and chapters have now reached over 400,000 downloads and are in 
the top 25% of downloads of Springer LNCS. We are extremely thankful to our sponsors 
for their commitment and support to the vision and mission of ICOST. 
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Real-time Pilates Posture Recognition System 
Using Deep Learning Model 


Hayoung Kim, Kyeong Teak Oh, Jaesuk Kim, Oyun Kwon, Junhwan Kwon, 
Jiwon Choi, and Sun K. Yoo) 


Department of Medical Engineering, Yonsei University, Seoul, South Korea 
sunkyoo@yuhs.ac 


Abstract. As the pandemic situation continues, many people exercise at home. 
Mat Pilates is a popular workout and effective core strengthening. Although many 
researchers have conducted pose recognition studies for exercise posture correc- 
tion, the study on Pilates exercise is only one case on static images. Therefore, 
for the purpose of exercise monitoring, we propose a real-time Pilates posture 
recognition system on a smartphone for exercise monitoring. We aimed to recog- 
nize 8 Pilates exercises—Bridge, Head roll-up, Hundred, Roll-up, Teaser, Plank, 
Thigh stretch, and Swan. First, the Blazepose model is used to extract body joint 
features. Then, we designed a deep neural network model that recognizes Pilates 
based on the extracted body features. It also measures the number of workouts, 
duration, and similarity to experts in video sequences. The precision, recall, and 
fl-score of the posture recognition model are 0.90, 0.87, and 0.88, respectively. 
The introduced application is expected to be used for exercise management at 
home. 


Keywords: Pilates - Pose recognition - Pose correction - Deep neural network - 
Real-time - Smartphone 


1 Introduction 


The interest in digital home training programs has increased since the outbreak of 
COVID-19 as they are accessible and cost-effective to exercise [1]. Among the home 
exercises, Pilates is a popular exercise that has recently become increasingly widespread 
in rehabilitation therapy [2]. Pilates was developed by Joseph Pilates of Germany and is 
a full-body exercise made for rehabilitating patients during World War I [3]. The Pilates 
mat exercise program is effective in treating chronic low back pain by evaluating core 
muscle thickness [4]. In addition, Pilates exercise without using any equipment is use- 
ful for improving respiratory functions and disease-related symptoms [5]. However, an 
injury may occur by repeating the wrong motion without experts. Therefore, a Pilates 
posture recognition system for home training is required. 

In computer vision, exercise training systems through human posture estimation are 
being developed to increase exercise effectiveness and prevent injury in various sports 
fields [6]. Dittakavi, B. [7] conducted a study to classify postures from fixed images using 
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probability techniques for Yoga, Pilates, and Kung Fu exercises and to explain which joint 
motion was incorrect. Wu, Y. [8] classified 45 yoga movements and created a model that 
gave points to the exercise movements without an expert. They extracted image features 
using a convolutional neural network (CNN) and trained the model using a unique con- 
trastive loss that combines the L2 norm with cosine similarity. Li, J. [9] presented a model 
for classifying and evaluating 117 yoga movements. Yoga movements from 22 subjects 
were measured with an RGB-D camera. At this time, 3D coordinates were corrected by 
taking pictures from both the front and side. In addition, a new Cascade 2S-AGGN (Cas- 
cade graph convolutional neural network for yoga pose classification and assessment) 
model was presented by constructing a graph convolutional neural network (GCN) in a 
hierarchical structure. Zhao, Z [10] also used GCN to classify three types of motion and 
presented a model for correcting posture. The expert’s motion and the subject’s motion 
were trained separately and evaluated the subject’s motion. Dynamic time warping is 
used to compensate for the difference in each person’s movement speed. In previous 
studies, there is one case of Pilates exercise [7] correction of static images. However, 
Pilates pose correction studies on video sequences are necessary since Pilates involves 
segmental movements of the spine. Therefore, we studied Pilates exercise correction on 
video sequences. 

The purpose of this study is to develop a real-time Pilates mat exercise recogni- 
tion system on a smartphone for exercise management at home. We developed a Pilates 
exercise posture classification model that automatically recognizes the 8 Pilates exer- 
cises. We even added a parameter measurement function for exercise monitoring. We 
finally developed a real-time posture recognition system on the smartphone for user 
convenience. 

“Methods” describe a data set of Pilates exercises, a posture recognition method, 
and real-time exercise monitoring capabilities. In the “Results”, the results of the Pilates 
posture recognition model and the real-time monitoring system are mentioned. In the 
“Discussion”, the results are discussed. In “Conclusion”, the conclusion is described. 


2 Methods 


2.1 Data Collection 


We have selected 8 Pilates exercises as follows: Bridge, Head roll-up, Hundred, Roll-up, 
Teaser, Thigh stretch, Plank, and Swan. The example of the selected Pilates exercises is 
shown in Fig. 1. 

The Pilates data were acquired with the camera facing the exercise mat and at a dis- 
tance of 2.5m. We used the front camera of the Galaxy 22 smartphone. The 8 movements 
are repeated about 5 times and recorded as videos. The image acquisition sampling rate 
is 30 fps. A total of 15 subjects of videos were acquired, and the sex ratio was 6 males 
and 9 females. 

We extracted features of 33 joints from the acquired videos using the Blazepose 
model [11] to train and test the Pilates recognition model. In many posture recognition 
studies, Blazepose was used because of its high accuracy and performance [12-14]. 
The 33 joint features are the location of the x, y, and z coordinates and the visibility 
indicating whether the joint is visible. 9 target postures were manually labeled for each 
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image of the video sequences. It includes 8 Pilates postures and an “unknown” class to 
distinguish it from the prescribed exercises. The entire movement from the initial to the 
final position is trained, and the information of one frame is input into the model. The 
15 subjects’ videos are split into 11 and 4 for training and test data. The total number of 
Pilates classification data is 247,203. The training data is 179,446, and the test data is 
67,757. The number of samples for each 8 Pilates postures is shown in Table 1. 


(a) (b) (c) (d) 
(e) (f) (g) (h) 


Fig. 1. Example of Pilates poses. (a) Bridge, (b) Head roll up, (c) Hundred, (d) Roll up, (e) Teaser, 
(£) Thigh stretch, (g) Plank, (h) Swan. 


Table 1. The number of Pilates postures recognition samples. 


Index Class Training samples Test samples 
0 Unknown 87,814 34,251 
1 Bridge 11,083 3,460 
2 Head roll up 7,596 2,915 
3 Hundred 18,763 7,533 
4 Roll up 11,566 4,392 
5 Teaser 7,574 3,299 
6 Thigh stretch 11,364 3,472 
7 Plank 12,206 3,826 
8 Swan 11,480 4,609 
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2.2 Pose Recognition Model 


The Pilates posture recognition model is designed to automatical classify 8 Pilates and 
unknown postures. We designed a simple deep neural network to deploy on an Android 
phone. The input size of the model is 1 x 132 because the x, y, z, and visibility values of 
33 joint points from on frame are entered. Three fully connected layers with 128, 64, and 
16 neurons are stacked. A dropout rate of 0.2 was used between each fully connected 
layer to prevent overfitting. Finally, a soft-max layer for classifying multiple classes was 
connected. The output of the model is probabilities of 9 poses. 


2.3 Real-Time Exercise Monitoring System 


We propose a Pilates posture recognition system to operate in real-time on the smart- 
phone. The sequence was designed to extract the number of exercises, exercise duration, 
and similarity with expert posture to monitor Pilates exercise. To operate the system on 
the smartphone, Mediapipe [15] framework was used. 


Real-Time System Architecture. The architecture of the posture recognition system 
is shown in Fig. 2. First, in the pose estimation section, the features of one person’s 33 
joints are estimated by the Blazepose model when the input image comes through the 
camera. In the next pose recognition section, the pose recognition model classifies 8 
Pilates exercises using the pose features. After then, since there is flickering in which 
the model predicts a different posture while predicting posture, a moving average filter is 
added that uses the average value of the last 10. Then, the pose counter & time measure 
section counts the number of exercises and measures the exercise duration. In the pose 
correction section, the similarity of the expert and current posture is calculated using 
joint features and recognition results. Finally, these exercise measurement parameters 
are combined in the draw overlay image section to generate an output image, and the 
result is displayed on the screen. 


Pose Estimation 
Blazepose 


Pose Recognition 


Pose Counter & Time Measurement 
& Correction 


Draw Overlay Image 


Fig. 2. Architecture of the Real-time Pose Monitoring System 


Recognition of the Up-and-down Movement. The up-and-down for one posture was 
recognized using the Euclidean distance differential between two specific joints. It is used 
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for the following function, posture counting and correction. In the Bridge pose, the hips 
move away from the ankles as they lift up and come closer as they lower down, recognized 
as Bridge-up by ankle-to-hip distance increase. Similarly, Head roll-up, Roll-up, and 
Teaser are identified by ear-to-knee distance decrease. Thigh stretch-up is recognized 
by hip-to-foot index distance increase. Plank and Swan up-and-down movements are 
detected using hip-to-elbow and ear-to-elbow distance differentials, respectively. 


Exercise Count and Time Measurement. Exercise parameters for Pilates postures are 
shown on the screen for users to monitor exercise counts and duration. The number of 
exercises and duration were determined through pose recognition results and recogni- 
tion of up-and-down posture. The Hundred Posture, a sustained position, was counted 
independently without distance comparison. For other postures, counts were measured 
when transitioning from up to down or vice versa. Every exercise time was measured 
by starting when each exercise was recognized and ending when the motion was not 
recognized. 


Pose Correction. Joint angles were used as feedback parameters for Pilates postures. 
To enhance workout effectiveness, it is needed to pay attention to correct posture. The 
similarity between the current and the expert’s posture was assessed by comparing angles. 
The method for calculating the angle of each joint is as follows. As shown in Fig. 3 (a), 
when there are three joint points X1, X2, and X3, the radian angle was obtained by 
Eq. (1). And radian angles were converted to degree units. 
d= cog ZR cos!” T 
yı — X2 Y3 = Y2 

Angles were compared to reference angles using 4 specific pairs, as depicted in Fig. 3 
(b). These pairs include the shoulder angle between the ears and hip, hip angle between 
shoulder and knee, knee angle between hip and ankle, and ankle angle between knee 
and foot index. Pilates is primarily a core strengthening exercise, and the arms play an 
auxiliary role in balancing, so the joint angles around the torso were selected. 

A weighted joint angle difference method was used to compare postures with experts. 
The expert posture that the reference for posture comparison was acquired from YouTube 
videos. One cycle of exercise motions to be corrected is each of the previously recognized 
up posture and down posture. The one-cycle of exercise motion was compared with 
the expert by calculating each section’s average joint angle difference divided into 10 
sections. Then the angle difference was multiplied by each joint’s confidence score. 
Calculate the weighted joint angle difference for all 10 cycles and 8 joints as Eq. (2). 
Joints with high confidence scores were weighted to have more influence on the similarity 
measurement, the idea of the weighted distance method [16]. Finally, the angle difference 
was converted to a similarity score by normalizing 180 degrees as Eq. (3). 


(1) 


1 IẹsP J ; ; 
A= 5 Eaval —Ar,), i e (1, 10),j € (1,8). (2) 
180 — Ag; 
Score = a (3) 


where A gir denotes a weighted joint angle difference. The i and j denote period and joint 
index, respectively. V; j is average visibility of the i-th period and j-th joint. Aj i denotes 
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subject’s average angle of the i-th period and j-th joint. Aj i denotes reference average 
angle of the i-th period and j-th joint. 


(a) (b) 


X, (X45 Y4) 


X3 (X3, Y3) 


X2 (X2; Ya) 


\— foot index 


Fig. 3. Pose Angle. (a) Joint angle calculation using two coordinates, (b) main angle used for 
posture comparison. 


3 Results 


3.1 Result of Pose Recognition Model 


We conducted an experiment with 67,757 test samples from 4 subjects. Performance 
metrics for evaluating posture classification models used Precision, Recall, and Fl-score 
[17], which are widely used in the performance evaluation of classification models. The 
formulas are (4), (5), and (6), respectively. 


ws TP 
Precision = ————. (4) 
TP + FP 
TP 
Recall = ——_. (5) 
TP + FN 


recall - precision 


(6) 


F1 — score = 2. 


recall + precision 


The precision, recall, and f1-score of the recognition model are 0.90, 0.87, and 0.84, 
respectively. The results of three metrics for 9 classes are shown in Table 2. 


3.2 Results of Real-Time Exercise Monitoring System 


Results of Exercise Monitoring System on Test Data. The exercise monitoring system 
was tested with videos of 4 subjects on the desktop CPU. These videos are the same 
data used to test the pose recognition model. Table 3 shows the exercise counts for each 
subject in the 4 test videos. The last image of the Pilates monitoring system for each 
subject video is shown in Fig. 4. In Fig. 4, the first line indicates the current posture’s 


Real-Time Pilates Posture Recognition System Using Deep Learning Model 


Table 2. Result of Pilates postures recognition model. 
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Index Class Precision Recall Fl-score 
0 Unknown 0.93 0.92 0.93 
1 Bridge 0.97 0.88 0.92 
2 Head roll-up 0.86 0.91 0.88 
3 Hundred 0.95 0.97 0.96 
4 Roll-up 0.92 0.84 0.88 
5 Teaser 0.86 0.75 0.80 
6 Thigh stretch 0.92 0.93 0.92 
7 Plank 0.83 0.99 0.90 
8 Swan 0.87 0.96 0.92 


count, duration, and the previous posture’s similarity score. Every 8 exercise’s count, 
duration, and similarity score were also displayed on the screen. Figure 5 shows an 
example of similarity score comparison in the Roll-up posture. In the Roll-up posture, 
subject 2’s score, who stood with his legs stable on the floor (Fig. 5 (a)), was 0.81 (Fig. 5 
(b)), but subject 3, who came up using the recoil of bending his knees (Fig. 5 (c)), scored 
0.77 (Fig. 5 (d)). 


Table 3. The number of Pilates exercises for test data. 


Pilates Exercise 


Bridge 


Subject 1 


Subject 2 


Subject 3 


Subject 4 


Head roll-up 


Hundred 


Roll-up 


Teaser 


Thigh stretch 


NI NIN NI] AJU 


Nini ni INJA 


NIN! NVuny~ nin 


DINDIAND\ Ni o;n 


Plank 


Swan 


Results of Real-time system on a smartphone. We checked the operation of the real- 
time Pilates exercise monitoring system on the Galaxy 22 smartphone. Model inference 
ran on the GPU, and the rest of the calculations ran on the CPU. A total of 8 exercises 
were performed twice, and the screen of the app was recorded. Table 5 shows the number 
of each Pilates exercises performed for test 1 and test 2. The last image of the application 
for each test is shown in Fig. 6. The number of exercises, time, and similarity score for 
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Fig. 4. The result of the Pilates posture monitoring system’s count, time, and similarity score on 
test data. (a) subject 1, (b) subject 2, (c) subject 3, (d) subject 4. 
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Fig. 5. Example of Roll-down posture score comparison. (a) Subject 2’s roll-up posture. The score 
is the previous roll-down posture, (b) Subject 2’s roll-down posture. The score is the previous roll- 
up posture, (c) Subject 3’s roll-up posture. The score is the previous roll-down posture, (d) Subject 


3’s roll-down posture. The score is the previous roll-up posture. 


each motion are displayed. Figure 7 shows examples of scores and count results for the 
Teaser and Plank postures in the two tests (Table 4). 
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Table 4. The repetition of Pilates exercises on the test data. 


Pilates Exercise Test 1 Test 2 
Bridge 5 5 
Head roll-up 5 5 
Hundred 2 2 
Roll-up 5 5 
Teaser 5 6 
Thigh stretch 5 5 
Plank 10 10 
Swan 5 5 
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Fig. 6. The result of the real-time Pilates exercise monitoring application on a smartphone. (a) 
test 1, (b) test 2. 


4 Discussion 


We conducted a pose recognition and correction on video sequences for home Pilates 
exercise monitoring. Since there is no open data set for Pilates exercise, datasets were 
newly acquired. 8 Pilates exercises were selected as easy for beginners to follow and 
good for back pain prevention, abdominal exercise, and stretching. 
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Fig. 7. Example results of Teaser and Plank in Tests 1 and 2 of the real-time Pilates exercise 
application. (a) Teaser up of test 1, (b) Teaser down of test 1, (c) Plank down of test 1, (d) Plank 
up of test 2, (e) Teaser up of test 2, (f£) Teaser down of test 2, (g) Plank down of test 2, (h) Plank 


up of test 2. 


Using the Blazepose model and a simple neural network, it showed recognition of 
8 Pilates exercises. Most errors in the pose recognition models occurred between the 
8 target exercises and the unknown class. There were also recognition errors in lying 
postures such as Hundred, Roll-up, Teaser, and prostrating postures between Swan and 
Plank since the entire movement was trained. Therefore, a moving average filter was used 
to prevent the flickering of class prediction in the middle of the video sequences. The 
number of exercises, exercise duration, and similarity score with experts were calculated 
using the pose recognition model’s prediction. Most count errors in the test videos are 
due to recognition errors in rollups and teasers (Fig. 4). We confirmed that all exercises 
counted well except head up, teaser, and plank on the smartphone. In both test! and 
test2, the 2 Head-ups were not counted because the head was not raised enough in the 
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head-up posture (Fig. 6). In testl, 1 Teaser was not counted (Fig. 6 (a)) because the 
subject fell so quickly as not recognized up-down (Fig. 7 (a), (b)). The remaining 4 
Teasers are counted, but the score is 0 because they did not exercise enough to compare 
the movements (Fig. 6 (a)). However, in test 2, all the Teasers are counted and scored 
as enough exercise (Fig. 7 (e), (f)). Meanwhile, In Test 1, all the Planks were counted 
(Fig. 6 (a)) as the postures were well recognized (Fig. 7 (c), (d)). In test 2, 5 Planks were 
not counted (Fig. 6 (b)) because they were recognized as unknown due to the wrong 
motion of lifting the hip (Fig. 7 (g), (h)). With the development of our proposed exercise 
monitoring app, we expect that users will be able to receive Pilates exercise feedback 
regardless of location [16]. 


Limitations and Future Work. Although there were some technical limitations, we 
have plans for improvement. Since the data was acquired on only direction, there is a 
limitation to posture recognition. It is necessary to acquire data from various angles. The 
errors in posture recognition not only affect count and time measurements but also pose 
correction. Therefore, our future plans involve developing recognition models using time 
series data to enhance performance. Additionally, we aim to implement Devanne, M. 
[18]’s method to provide detailed feedback on specific body parts and overall posture. 


5 Conclusion 


In this paper, we propose a Pilates exercises monitoring system on a smartphone in real- 
time. First, we acquired video sequences for 8 Pilates postures. Pilates postures were 
recognized using a deep learning model on video sequences. In addition, exercise count 
and time measurement function for measuring exercise volume were added. We also 
proposed a weighted joint angle distance method that measures the angle and movement 
of major joints and compares posture with experts. It is expected that Pilates exercises 
can be corrected at home without experts. 
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Abstract. The presented platform architecture and deployed implementation in 
real-life clinical and home care settings on four Amyotrophic Lateral Sclerosis 
(ALS) and Multiple Sclerosis (MS) study sites, integrates the novel working tools 
for improved disease management with the initial releases of the AI models for 
disease monitoring. The described robust industry-standard scalable platform is 
to be a referent example of the integration approach based on loose coupling APIs 
and industry open standard human-readable and language-independent interface 
specifications, and its successful baseline implementation for further upcoming 
releases of additional and more advanced AI models and supporting pipelines (such 
as for ALS and MS progression prediction, patient stratification, and ambiental 
exposure modelling) in the following development. 


Keywords: amyotrophic lateral sclerosis - ALS - multiple sclerosis - MS - 
clinical disability function - ALSFRS - extended disability status -EDSS - REST 
services - architecture - AI models - disease progression - prediction - patient 
stratification - relapse - Big Data - pre-processing - regularization - daily living 
activities - exploratory analytics - neurodegenerative - wearable sensors - IoT 
integration 


1 Introduction and Scientific/Technological Background 


The BRAINTEASER Project (BRinging Artificial INTElligence home for a better care 
of Amyotrophic lateral Sclerosis and multiple sclERosis), funded from the European 
Commission Horizon 2020 programme grant until the end of 2024, integrates heteroge- 
neous societal, environmental, health, and lifestyle/habitual data from diverse sources, 
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developing patient stratification and disease progression AI models and applicative tools 
for improving disease management and ubiquitous monitoring and care delivery for ALS 
and MS patients (and assistance to informal caregivers). 

Both are very complex chronic progressively degenerative fatal neurological dis- 
eases significantly disrupting the quality of life of the patients and their families, with 
notable differences in clinical picture, evolution, prognosis and therapies, but also many 
similarities in modelling and care/intervention delivery contexts (both in clinical and 
outpatient settings). 

The initial releases of the novel interactive applicative tools for disease management 
and monitoring, currently in use in real-life settings in four clinical study validation 
sites (Lisbon, Madrid, Pavia and Turin) of the Project, are described with main features 
and functionalities in [1], including also some of the basics of the underlying back-end 
platform architecture and implementation. These tools and platform middleware services 
are enabling and supporting the first crucial step in the main overall identified and 
modelled BRAINTEASER data and process flows — continuous acquisition, ingestion, 
integration and storage of: 


e detailed retrospective and prospective clinical datasets, with the prospective ones 
collected in the course of the Project studies being additionally complemented and 
augmented by 

e comprehensive heterogeneous personal health, activity, lifestyle, habit- 
ual/behavioural, and environmental data, collected using: 


e digitalized instruments and questionnaires for ALS and MS (and comorbidities) 
clinical evaluation and remote disease progress assessment, both standardized and 
in common practice (like ALSFRS-R or EDSS), as well as some innovative and 
evolving ones (like Awaji-Shima Consensus or Gold Coast diagnostic criteria for 
ALS), and 

e commonly available sensing/IoT devices, mainly Garmin smartwatches, and 
portable and fixed air quality and atmospheric para-meters sensing devices (like 
Atmotube PRO! and PurpleAir (Classic) PA-II’). 


The collected data have driven the development of Artificial Intelligence (AT) models 
able to address the needs of precision medicine, enabling early risk prediction of dis- 
ease fast progression and adverse events. During the previous yearly period the intense 
development and evaluation of AI models in the Project (including relevant Open Science 
efforts co-organized by the Project? [2]) have further resulted with the first releases of 
the model routines, tested and delivered ready for integration. 

These generate not only the main envisioned final model outputs - such as predic- 
tions of probabilities or timeframes of occurrence of key disease progression events, 
like MS relapse or introduction of NIMV (Non-Invasive Mechanical Ventilation) or 
PEG (Percutaneous Endoscopic Gastrostomy) treatment for a patient - but also include 


1 https://atmotube.com/atmotube-pro. 

2 https://www2.purpleair.com/products/purpleair-pa-ii. 

? https://brainteaser.health/open-evaluation-challenges. 

4 https://brainteaser.health/open-evaluation-challenges/idpp-2022. 
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the pre-processing or re-calibration model routines within the models, generating inter- 
mediate summary aggregated or transformed values from raw data inputs fetched from 
the unified platform Data Store, and passing the results back again to be persisted in the 
Data Store and re-used mostly for provision to the consuming applications exposed to 
the targeted end users, and as inputs to other following routines further in the disease 
progression prediction and patient stratification model pipelines. Such pre-processing 
routines, having been released first as initial steps in the ad-hoc AI processing pipelines, 
are actually turning out to be the most demanding ones from the integration perspective, 
as they require more frequent periodic invocation and execution (up to several times 
daily), and generate much higher throughput and consumption of data exchanged with 
the core platform services than the routines for actual disease progression prediction and 
patient stratification (invoked once in weeks or even months, and relying on already pre- 
processed results as more compacted inputs). Main algorithm types exploited in these 
pre-processing, feature extraction/selection and dimensionality reduction routines are 
Bayesian filtering and smoothing, retiming, or oximetry digital biomarkers evaluation 
(pobm? package), while the further AI models for continuous disease monitoring and 
progression prediction yet to be deployed exploit survival analysis algorithms like Cox 
proportional hazards, supported by methods like forward-recursive feature selection, 
and others elaborated fully and in detail in related publications like [10]. 

This paper describes architecture and implementation of the core BRAINTEASER 
platform back end and services tier integrating and supporting the mentioned released AI 
model routines in operation (as well as other recently developed or advanced supported 
features of the data feeding and consuming applications or modules coupled to the plat- 
form, presented summarily as a reminder on the general ecosystem and flows overview 
on Fig. 1 below). The general integration approach, as described in the following sec- 
tions, is the same for both abovementioned types of AI model routines for now, with 
eventual specific critically performance-dependent alternative pathways and design pat- 
terns supporting tighter coupling or more extensive query stream parallelism having also 
been developed in reserve, and used for some scenarios of environmental/ambiental data 
processing, as described in [8]. Pilot demonstration and validation phase of the Project 
(initially focused on ubiquitous personal data collection, cleaning and integration at this 
stage, as mentioned) has just started a couple of months ago at the time of writing of 
this paper, with still a limited number of recruited study subjects and collected feed- 
back and data on the usage. After at least a further semester of increased recruitment and 
more intensive continuous usage of the BRAINTEASER platform applications deployed 
towards the end-users, there will hopefully be sufficient data and statistics on the usage 
of the platform for at least an elementary sound and substantial analysis of the overall 
results and performance of the deployed implementation of the architecture to comple- 
ment and expand on the work reported here, requiring more space than available in this 
short conference publication format, possibly in an evolved derived journal article. 


E https://pypi.org/project/pobm. 
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2 Architectural Overview 


The diagram on Fig. | below provides a broad overview of the overall envisaged BRAIN- 
TEASER Data and ICT Tools ecosystem with main data flows and structural breakdown 
of data consuming or sourcing components and modules (more detailed in particular of 
the services tier in the biggest frame top right, crucial for integration), and the logo icons 
next to main modules/tiers to be described additionally denoting the system infrastruc- 
ture language or platform stack chosen for development and deployment ( @)- Python, 
M- PostgreSQL, <- Java). 

The presented schema is an evolution and expansion of the similar initial one provided 
as Fig. 7 in [1], and as noted in that referenced article, some of the platform functionalities 
and architecture build and extend upon the related efforts and outputs from preceding 
and parallel related projects, specifically from PULSE* (partially described in [3, 5, 7]), 
NEVERMIND’ (in [6]), and PERISCOPE? (in [4]). 
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Fig. 1. Updated BRAINTEASER ecosystem architecture with main data sources, consumers and 
flows, and detailed service tier breakdown. 


RESTful APIs are the main interfacing and architectural approach on the back-end 
tier, with secure communication between the web services enhanced by industry standard 
JWT? (JSON Web Token) lightweight encrypted encapsulation of request and response 
payloads. This extends also to the implementation and deployment of the AI models and 
routines themselves, with common unified human-readable and understandable inter- 
faces specification based on JSONs, agnostic of the language the wrapped underlying 


6 https://www.project-pulse.eu. 

7 https://www.nevermindproject.org. 
3 https://periscopeproject.eu. 

? https://jwt.io. 
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logic is written in, being strongly preferred across the complete services and tools ecosys- 
tem. Some of the additional key benefits of this for facilitated maintenance, scalability 
and sustainability of the platform beyond the Project developments are: 


e OpenAPI (3.x) Specification (OAS) compliance, with ample open and available sup- 
port and standardized toolsets for easier and semi-automated generation and mainte- 
nance of API specification and documentation live online (currently on Swagger!’ at 
https://brainteaser. belit.co.rs/gateway/swagger-ui/index.html, with relevant example 
provided on Fig. 2 below), as well as for API testing, mocking, and overall lifecycle 
governance and scaling. 

Collections feature!! of the Postman REST API platform client has also been 
extensively used for building and maintaining request sets for practical integration 
testing of intra-service communication and data flows, and scheduled periodic batch 
server jobs, needed for execution management of most AI model routines, are also 
easier to consolidate and manage with uniform service calls (though still specific to 
application server/container or hosting server OS). 


e the encapsulation of AI model and preprocessing routines (commonly written by data 
scientists as Python or R functions) within standard web service endpoints, similar to 
the interfacing (Java-based) invoking and managing ones of the core platform, also 
unifies and simplifies the deployment and CI/CD (Continuous Integration/Continuous 
Delivery) pipelines across the platform, including deployment containerization and 
scaling (with tools like Docker). 

Concrete specifics of this web application server/container “wrapping” imple- 
mentation for Python model routines are provided in the following section. 


Development practice at this stage is to have each specific thematic domain set of AI 
models (for disease monitoring, progression prediction, patient stratification...) encap- 
sulated in a dedicated wrapping microservice as it gets completed and delivered for 
platform integration (Fig. 3). Later towards the release of the overall integrated ecosys- 
tem, refactoring for optimal modularity can be performed, possibly merging some of the 
microservices (or most of them, into a practical modular monolith architecture), accord- 
ing to the results of the continuous models screening, in-silico simulation, evaluation, 
and improvement pipeline in the scope of the Project WP4, and according to the finally 
identified performance requirements and constraints. 

The overall back-end service tier is similarly structured, mainly based on loosely 
coupled REST microservices [9], but with some practical trade-offs towards modular 
monolith, and the separation between domain-specific and infrastructural/utility orthog- 
onal logic. The service packages and sets specific to the BRAINTEASER main thematic 
domains logic (supporting features and functionalities related to patients, caregivers, dis- 
eases & comorbidities, IoT devices and data in the system, etc.), presented as horizontal 
rectangles in the main top right frame on Fig. 1, are grouped into a couple of subpro- 
jects in development, and are currently using just three separate schemas/tablespaces 
in the Data Store underneath, divided at this stage mainly according to non-functional 


19 https://swagger.io. 
i https://www.postman.com/collection. 
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= = On-demand processing of acquired wearables’ (Garmin) data through 
POST /device/data/processing/process WP5 (Python) pipeline, for a given user code and starting date. és 


No parameters 


Request body ™*"""" application/json {v 


Example Value Schema 


"access_code": 
“start_date": 


“device_installation_id": 
"db_id": 


Responses 
Code Description Links 
200 No links 


Data successfully processed and stored 


Media type 


application/json {v 


Controls Accept header. 


Example Value | Schema 


Fig. 2. Example of testable specification/documentation of a platform API endpoint for on- 
demand invocation of a data pre-processing method generating inputs for the AI models for 
continuum disease monitoring (developed within the Work Package 5 (WP5) of the Project). 


requirements — one containing highly sensitive potentially personally-identifying data 
(descriptive reported symptoms, specific socio-demographic and profile data...) kept 
protected fully encrypted in the database and throughout all handling in the system, 
other with the “regular” non-protected or already de-identified data, and the third for 
metadata. As the ingested data volume increases, the first two will likely separate to at 
least another additional dedicated to most bulky IoT sensed measurements data. 
Microservices with separate underlying schemas supporting the infrastructural func- 
tionalities orthogonal to the domain logic (access control, authentication, security, utili- 
ties...) are represented as rectangles with vertical labels on Fig. 1, with the API Gateway 
package implementing the design pattern for basic service orchestration. Subsystems are 
hidden behind the Gateway façade service, acting not only as a proxy to those domain 
services but also validating requests (in terms of tokens, basic structure, sequencing...) 
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referent data 


referent data 
extraction 
and 
preparation | 


Mode, 


Fig. 3. Generic pipeline flows and microservice encapsulations of the integration of AI model 
and pre-processing methods with the core BRAINTEASER Platform services and Data Store 


and documenting the specifications of all services through Swagger. Gateway also imple- 
ments composition of calls for operations requiring calls to multiple services (or mul- 
tiple calls to a single service), which is preferred to complex network of direct calls 
between sub-services. Code for request and response model, and endpoint definitions 
(URI, method, and docs) of sub-services are shared with Gateway by Git submodules in 
CI/CD. 

Some additional most prominent employed service design patterns, mainly for the 
data collection, fusion, and provision to the applicative tools for disease monitoring and 
management, are described in [8]. 


3 Implementation 


Service tier core is currently implemented on the robust heavy-duty industry standard and 
proven Enterprise Java (Amazon Coretto 17 LTS!) web technology stack, leveraging 
Spring Boot!? framework 2.6.4 with all the functional programming and REST APIs 
support. Upgrade to the next Java LTS (Long-Term Support) version, expected to be Java 
21 released around September 2023, is planned and being prepared for jOOQ!* (Java 
Object Oriented Querying) framework is used for object-relational mapping (ORM), 
and all is deployed and running on the Apache Tomcat 9.x web application server. 

The industry-standard and common jUnit 5 framework!» is used for unit test gen- 
eration, with REST Assured!° for baseline Java REST services semi-automated testing 


E https://docs.aws.amazon.com/corretto/latest/corretto- 17-ug/what-is-corretto-17.html. 
13 https://spring.io/projects/spring-boot. 

4 https://www.jooq.org. 

D https://junit.org/junit5. 

16 https://rest-assured.io. 
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purposes, along with other extensive REST API development and lifecycle management 
tools specified above. 

Other common and mostly newer open-source alternatives, like for example a com- 
plete Python-based stack covering also the AI models implementation, or JavaScript 
server-side implementations, still do not offer as comprehensive and robust support and 
performance in REST API services implementation, both object-oriented and functional 
programming, and cross-platform (heavy-duty web and mobile applications being the 
key BRAINTEASER tools for the end users) development. 

GitLab serves as the code repository and version control system, as well as for CI/CD 
pipelines and control (except for the data tier, where Red Gate Flyway!’ tool is used for 
database versioning management and migration/replication control). 

Parallel Python (3.x.x) runtime is hosting the integrated AI model methods for sen- 
sory data pre-processing and disease monitoring, encapsulated in web service pipelines 
as described above, using the RESTful API implementation via the lightweight Flask!® 
web framework deployed along with the Java-based core services stack on the platform 
back-end server cloud, with SQL Alchemy!’ being the optional ORM framework in 
the Python-based stack. Principally, similar language-agnostic implementation would 
be viable for all AI models envisioned to be developed in the Project (more details in 
the next section with conclusions), supporting loosely coupled APIs for seamless scal- 
ing or significant changes (in cases like e.g., specific model routines getting completely 
rewritten and implemented in R or Julia, REST-based interfacing and service invocation 
and deployment control should remain unchanged, if a corresponding frameworks like 
Plumber or Genie are used instead of Flask). 

Unified JSON-based communication across the platform also provides for some 
advantages on the data tier — implemented as PostgreSQL hybrid-relational Data Store, 
it features extensive support for JSON and binary JSON data types, querying, index- 
ing and optimization. Consequently a lot of data that are natively structured in JSON 
documents as collected or generated in the ecosystem (evolving generic questionnaires, 
service configurations, intervention and gamified content...) are stored and queried in 
JSON format in the database, and fully deserialized into relational model entities only 
when necessary for main data consumption and performance criteria, or the structure 
evidently standardized and fixed in the long-term or permanently (like for the standard- 
ized questionnaire instruments exemplified in Sect. | above, or data model structures 
compliant with the relevant architectural standards in healthcare IT, mainly ISO/CEN 
13606, openEHR, and HL7 FHIR...). This implementation has in development and 
deployment practice experiences by now shown equal or comparative performance in 
handling the mentioned document-structured data as using dedicated document-oriented 
databases like MongoDB, while at the same time retaining advantages of native rela- 
tional data support (most of the data handled by the overall platform are relational by 
nature) or more comprehensive and robust transaction control (nesting, cascading), all in 
a single unified data store managing the complete heterogeneity of data. PostgreSQL has 
also shown satisfactory overall performance with the terabyte-level volumes of sensed 


a https://flywaydb.org. 
18 https://palletsprojects.com/p/flask. 
13 https://www.sqlalchemy.org. 
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IoT data (billion-record tables) expected to be collected by the end of the Project, with 
proper indexing and query optimization, and using multiple available and well supported 
dedicated extensions (for scalability, time-series data management, etc.). 

Leveraging the JSON-LD” format for Linked Data is also convenient over the com- 
plete uniform JSON platform interfacing and communication, for seamless expected 
upcoming semantic integration, requiring minimal overhead efforts and data model 
changes, with the BRAINTEASER Semantic Cloud [2] that has been developed and 
evolved during the two initial Project years mostly independently from the described 
platform. A referent similar semantic integration example is shortly described in Sect. 4.3 
in [4]. 


4 Conclusion and Further Development and Evolution 


The presented platform architecture and deployed implementation in real-life clinical and 
home care settings on four BRAINTEASER study sites, integrating the novel working 
tools for improved ALS and MS monitoring and management released last year with 
the initial releases of the AI models for disease monitoring (and the supporting data 
pre-processing pipeline). 

This integration of two key types of targeted ICT outputs of the BRAINTEASER 
Project through the described robust industry-standard scalable platform is to be a ref- 
erent example of the integration approach based on loose coupling APIs and industry 
open standard human-readable and language-independent interface specifications, and 
its successful baseline implementation for further upcoming releases of additional and 
more advanced AI models and supporting pipelines (such as for ALS and MS progres- 
sion prediction, patient stratification, and ambiental exposure modelling) in development 
until the end of 2024. 


Acknowledgement. The presented work and developments are being funded by the European 
Commission Horizon 2020 programme grant agreement GA101017598, in the scope of the 
BRAINTEASER project. 
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Abstract. Machine learning (ML) algorithms have become popular in 
recent years and have found increasing utility in the field of medical imag- 
ing, specifically in positron emission tomography (PET) imaging. The 
interest in ML in PET imaging for the study of neurodegenerative dis- 
eases stems from the potential of these techniques to analyze and predict 
the physiological parameters of biomarkers such as the total volume of 
distribution (V+) in the organ or a structure of the organ to be explored. 
In this paper, we investigated whether the V; of [‘SF]-FEPPA radio- 
tracer, an indicator of neuroinflammation, could be estimated directly in 
a non-invasive way, given the activity of the radiotracer in brain tissue. 
The study used several regression models to predict the ['SF]-FEPPA Vt 
in different brain regions where 31 regions of interest were defined for 
each of 24 patients with Parkinson disease and 20 healthy subjects, and 
were used to train four tree-based regression models. The predicted and 
reference values were compared by Bland-Altman analysis and regression 
model’s performance was evaluated by the mean absolute error (MAE). 
The best result was obtained by the XGBoost model with a MAE of 2.6. 
Bland-Altman analysis results indicate that predicted V, are in average 
very close to the reference with a bias of 0.23 F 2.82. Significant main 
effect of genotype on ['SF]-FEPPA in both caudate and putamen have 
been preserved by predicted Vt values (p < 0.05). The results of paired 
t-test indicate that the difference between predicted and reference V; is 
not statistically significant in 6 out of 8 groups. The proposed algorithms 
provide a non-invasive and efficient tool to predict ['SF]-FEPPA V, val- 
ues, a hallmark of neuroinflammation that is believed to be a potential 
trigger for Parkinson’s disease development. 
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1 Introduction and Related Work 


Machine learning (ML) is on the verge of revolutionizing medical diagnosis, espe- 
cially in imaging-based specialties such as PET. This new discipline is bringing 
a wealth of innovations to the analysis of large datasets in critical clinical studies 
like those focused on neurodegenerative diseases. In this article, we propose a 
set of ML-based regression models to reproduce [!8F|-FEPPA V, values, which 
are a neuroinflammation hallmark, in a non-invasive way. 

Neuroinflammation is a complex process involving the activation of immune 
cells within the central nervous system (CNS) in response to injury or infection 
[1]. While it is normal and essential for a neuroinflammation to occur in the brain 
as a response to previous triggers, it has been proven that an excessive or chronic 
inflammation may lead to the development and progression of neurodegenerative 
diseases such as Parkinson’s disease (PD) [2] and lead consequently to a loss of 
dopamine levels in the striatum region of the brain, more specifically in caudate 
and putamen regions. 

Many factors contribute to this loss, including genetic factors [3], environ- 
mental factors [4], age [5], etc. Furthermore, several studies have shown that the 
activation of inflammatory pathways in the brain can contribute to the death of 
more dopamine neurons and amplify the motor symptoms of PD [6]. The inflam- 
mation in PD is actually characterized by the activation of particular glial cells 
in CNS, called microglia [7]. Recent studies have shown that a chronic activa- 
tion of these cells can become harmful to the CNS [8]. Based on that, researchers 
have targeted neuroinflammation as a hallmark feature for PD’s disease, as well 
as for other neurodegenerative diseases such as Alzheimer’s disease [9], Hunt- 
ington’s disease [10], etc. Tracking neuroinflammation can potentially provide 
valuable insights into the mechanisms underlying these diseases, in addition to 
an accurate and earlier detection and diagnosis, enabling earlier interventions 
and treatments [11]. PET uses a special scanner to detect radiation emitted by 
a small amount of radioactive substance called radiotracer or radiopharmaceuti- 
cal, that has been injected into the body. The used radiotracer binds to markers 
of inflammation or other physiological processes that are associated with these 
diseases. 

Researches have proved that measuring microglia activation can be performed 
through quantifying a protein called translocator 18 kDa protein (TSPO). The 
expression of TSPO is upregulated when microglia are activated in response to 
injury or other stimuli in the brain, hence the interest of researchers to develop 
TSPO radiotracers [12,13]. ['SF]-FEPPA is one of the newest second genera- 
tion TSPO PET radiotracer with greater affinity for its target. However, the 
quantification of its distribution in the brain cells requires the determination of 
a metabolite-corrected arterial input function (AIF), which is practically done 
through arterial cannulation. Although the risk related to arterial cannulation is 
low, it represents an invasive and logistically demanding procedure. Besides, the 
discomfort caused by this procedure often discourages subjects from participat- 
ing in PET studies. Given these considerations, much studies have been carried 
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out to obviate the need for arterial cannulation such as Population-Based Input 
Function (PBIF) [14-17] and Image Derived Input Function (IDIF) [18-21]. 

Recent studies have explored the use of machine learning (ML) approaches 
for estimating AIF [22-24]. While the use of these approaches has not been 
extensively investigated, recent studies have reported promising results. As the 
aforementioned methods, MLIF has different challenges to be addressed like 
the high dependency of its results on the quantity and quality of the available 
training data. 

To the best of our knowledge, this is the first study to investigate the estima- 
tion of pharmacokinetic parameters directly from the given Time Activity Curves 
(TACs) of different brain regions and, hence, avoid any use of AIF. By using ML 
models, we were able to give an approximate value of V+, in a non-invasive way. 
Our results provide a novel perspective on PET images quantification. 

Our paper addresses leveraging appropriate ML techniques to predict, in a 
non-invasive manner, the total volume of distribution (V+) given the regional 
TACs. The data used in this study was acquired with [‘8F|-FEPPA radiotracer. 
We organized our paper as follows: in the first section, we will give an overview 
of the dataset acquisition, the reference V; estimation and the methods we used 
for Vt prediction. Next, we will discuss the results obtained from our ML models. 
In the conclusion, we will suggest some directions for future work. 


2 Methods 


In this section, we present details on features that have been used as input to our 
models, including data acquisition, region of interest (ROI) delineation, TACs 
generation, input function used in the quantification of PET data previously 
published in [25] and the genotype of the subjects included in the analysis. The 
training data contain relevant information about the biochemical transforma- 
tions of the tracer in each of the brain regions and the genotype of the studied 
subject, as well as the correct response to the [!SF]-FEPPA V+. 


2.1 Data Acquisition 


Twenty four subjects with Parkinson’s Disease and twenty healthy controls 
underwent an ['8F]-FEPPA PET and magnetic resonance imaging scan. After 
radiotracer administration into the body of the patient and PET data acquisi- 
tion, the collected PET raw data were reconstructed into images using ordered 
subset expectation maximization with point spread function (OSEM+PSF) 
reconstruction [26]. 


2.2  ROI-Based Time Activity Curve Generation 


MRI images for all the subjects were acquired for co-registration with the cor- 
responding PET images and the anatomical delineation of the 31 ROIs. The 
ROI template is transferred to the PET image space to extract the time activity 
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curve for each ROI. The TACs are graphical representations of how radioac- 
tive tracers distribute and accumulate within tissues over time. In our study, 
dynamical series of images of [!8F]-FEPPA PET have been visually checked for 
head-motion and corrected using frame-by-frame realignment [25]. 


2.3 Input Function Measurement 


AIF is determined during the PET scanning by gathering blood samples at 
discrete time points from the subject’s radial artery and measuring the con- 
centration of the radioactive compound in every sample. Arterial blood was 
taken continuously at a rate 2.5 mL/min for the first 22.5 min after radioligand 
injection and the blood radioactivity levels were measured using an automatic 
blood sampling system (Model PBS-101 from Veenstra Instruments, Joure, The 
Netherlands). In addition, 4 to 8 ml manual arterial blood samples were obtained 
at 2.5, 7, 12, 15, 30, 45, 60, 90, and 120 min relative to time of injection. A bi- 
exponential function was used to fit the blood-to-plasma ratios. A Hill function 
was used to fit the percentage of unmetabolized radioligand. The dispersion 
effect was modeled as to the convolution with a monoexponential with disper- 
sion coefficient of 16s and corrected with iterative deconvolution [27]. 


2.4 Polymorphism Genotyping 


The quantitative interpretations of [‘SF]-FEPPA are impacted by the large inter- 
individual variability in binding affinity, which displays a trimodal distribution 
compatible with a co-dominant genetic trait [28]. Study of TSPO polymorphism 
explained the heterogeneity in binding potential by the difference in the affinity 
of the second-generation PET ligands for this protein. [!SF]-FEPPA radiotracers 
bind TSPO in brain tissue from different subjects in one of three ways: high- 
affinity binders, mixed affinity binders, and low-affinity binders (HABs, MABs 
and LABs). The transport rate of radiotracer is 1.5 to 2-fold higher in HABs 
than MABs and 4-fold higher in HABs than LABs. Since LABs are very rare, 
we limited our data collection to the two groups HABs and MABs only. More 
insights on TSPO polymorphism can be found in [29]. 


2.5 Kinetic Analysis 


Kinetic modeling is a mathematical approach used in PET imaging to quantify 
the pharmacokinetics of a radiotracer in various tissues. In this study, we used 
the 2-compartmental model (2-TCM) [30] to fit our data. This model assumes 
that the radiotracer in the tissue compartment can be either specifically bound 
to the target receptor (specifically bound compartment) or it can be free (non- 
specifically bound compartment). The kinetics of tracer uptake have thus been 
modeled mathematically through differential equations describing the exchange 
rate of tracer concentrations among compartments in function of time as follows 
(1): 
dCy 


-z 1 Op(t) — (ha + ks)C1 (t) + kaC2(t) (1) 
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T = ksCi(t) — kaCo(t) (2) 


where: 

Cı is the tracer concentration in the non-displaceable compartment of the 
tissue (free and nonspecifically bound), Cp is the tracer concentration in the 
plasma also known as Arterial Input Function (AIF), C2 is the tracer concentra- 
tion bound to the target receptors, K; is the rate constant for transfer of tracer 
from plasma to the tissue, k2 is the rate constants for transfer of tracer from 
tissue to plasma, ka and k4 are the rate constants for transfer of tracer from 
the non-displaceable compartment to the specific binding compartment of the 
tissue and vice versa, respectively. Knowing the tissue compartment comprises 
two different states of binding (non-displaceable + specific binding), the tissue 
concentration, C;(t), is equal to the sum of the two states 


C(t) = C1 (t) + Co(t) (3) 
Using the aforementioned differential equations, C;(t) can be defined as follows: 


Kı 


ARE 


[(ks + ka — bi) e™* + (bz — ks — ka) e™] 8 Cpt) (4) 


where bı and bz expressions are 


bio = 5 |(k2 + ks + k4) F V (ke H kg + ka)? — 4kokg (5) 


and ® denotes the mathematical convolution. In this study, we have a particular 
interest in deriving quantitative information about the total volume of distribu- 
tion of the radiotracer as this kinetic parameter reflects the overall density and 
distribution of the TSPO receptors in the brain. Precisely, V, represents the ratio 
of the tracer amount in the target tissue at equilibrium to the amount of tracer 
in the plasma at the same time point. Mathematically, V can be expressed in 
function of model rate constants as follows: 


V; = Kyi /k (1 + k3/ka) (6) 


In TSPO studies, a higher V, indicates a greater amount of tracer binding to 
the target protein, suggesting a higher level of neuroinflammation in the tissue 
[31]. 


2.6 Estimation of Reference Total Volume of Distribution 


An estimate of V, values was derived using the kinetic modeling tool of PMOD 
(https://www.pmod.com/web/). We utilized the blood TACs, plasma TACs, 
and regional TACs of the brain for each subject in our dataset to fit a reversible 
2-TCM model, enabling us to estimate Vt values. 
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2.7 Total Volume of Distribution Prediction 


After estimating the reference V, values corresponding to each brain region and 
for each subject, we built our dataset by concatenating the TACs of different 
subjects. Initially, the TAC file of each subject contains the different intervals 
of scans, defined by the start and end time of the scan, and the corresponding 
radiotracer concentration at this interval, with respect to each brain region. On 
average, we have 31 ROIs for each subject. Then, we added to our dataset 3 
categorical variables specifying the brain ROI, the genotype (HAB or MAB) 
and the health status (Healthy Control or Parkinson). For V; prediction, a total 
of 44 subjects were included for the purpose of establishing predictive mod- 
els using ML. Our predictive models are tree-based regression models, which 
use decision trees to predict continuous numerical values. The rationale behind 
choosing these approaches for our regression problem is their ability to handle 
the mixture of categorical and continuous variables as inputs, and to capture the 
non-linear relationships between variables. To find the optimal hyperparameters 
for our predictive models, a grid search approach was employed. Rather than 
dividing the available data into separate training and testing groups, 10-fold 
cross validation was utilized and the average performance was recorded. 


2.8 Total Volume of Distribution Evaluation 


To evaluate our models, the predicted V, values were compared with the ones 
estimated by kinetic modeling and denoted as reference V+, for each region of 
interest, by the mean absolute error (MAE), 


re x 
MAE = — Vti — Vti 
22 | (7) 


We also used Bland-Altman as a graphical method, to compare the predicted 
values of our ML models against the reference values, and asses the level of agree- 
ment between them. By plotting the difference between V; and V; against their 
mean, it will help to identify any systematic bias between the two measurements, 
as well as the range of differences and outliers. 


3 Results and Discussion 


Results from comparisons between the reference and the predicted V; in terms 
of MAE are summarized in Table 1. As shown in this table, all the tree models 
have predicted V, with a mean absolute error ranging between 2.62 and 3, which 
is within an acceptable range of error for our particular problem, considering the 
median value of reference V+. These results indicate that our models are able 
to predict the target variable with reasonable accuracy and provide a good fit 
to the data. After evaluating the performance of the four tree-based models, we 
found that XGBoost outperformed the other models in terms of its evaluation 


32 O. Gassara et al. 


Table 1. Summary of Mean Absolute Error values for the selected machine learning 
models 


Model MAE 


DecisionTreeRegressor 2.99 


RandomForestRegressor 2.85 


GradientBoostingRegressor | 2.74 
XGBRegressor 2.62 


metrics. It specifically achieved a lower MAE, equal to 2.62 compared to other 
models, indicating that it has a better overall performance and it can predict 
the target variable with greater precision. For this model, we investigated the 
relative importance of each feature in predicting the target variable. Based on 
the importance scores of XGBoost model, we can conclude that the tracer uptake 
concentration for the first 7 timepoints are the most influential features in making 
V+ predictions. In other words, these features are the most utilized by XGBoost 
in a split decision, while creating its decision trees. 

In order to provide a more comprehensive evaluation of the performance of 
XGBoost, we used the bland-altman plots as showed in Fig. 1. Central and outer 
dashed lines indicate mean value and mean + 1.96 SD. It is important to mention 
that bland-altman is displaying the difference and average between the reference 
and predicted V, for all the tissues of test data subjects. Figure 1 shows that the 
mean difference is 0 with a bias of 0.23 + 2.82, among brain tissues, and 1.96 
SD interval equal to —5.30 and +5.77, which is believed to be a tolerable result 
for a first attempt of predicting V+ directly from tracer uptake concentration in 
brain tissues. Furthermore, we can observe that the majority of the data points 
fell within the limits of agreement, reflecting an overall good agreement between 
the reference and predicted values of V+. 

To further validate the results given by XGBoost, we investigated the ability 
of our model to highlight the genetic subgroup effects on TSPO binding. These 
effects have been previously explored in [25]. We are interested in reproducing 
these results, as we are analyzing the same dataset. In this part of our study, 
we will be focusing on putamen and caudate nucleus regions, as these regions 
are critically involved in the pathophysiology of Parkinson’s disease. Given the 
two possible studied genetic subgroups (HAB or MAB) and health status (HC 
or PD), we can distinguish between 4 groups as shown in Fig.2. A total of 
12 subjects were substracted from the initial data, 3 subjects from each group 
(HAB-HC, HAB-PD, MAB-HC and MAB-PD) to constitute the test data and 
we trained XGBoost model on the remaining data. The results showing the effect 
of genotype (MAB or HAB) on estimated V, and predicted V, for the caudate 
nucleus and putamen are illustrated in Fig. 2. This figure suggests that we can 
preserve the same effect of genotype revealed by the kinetic modeling estimated 
V+, with the exception of caudate V+ values of one group, which is the PD group. 
According to ML predicted V+, there is no significant difference in caudate V; 
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Fig. 1. Bland-Altman plots of predicted and reference total volume of distribution Vi 
using XGBoost model for all brain tissue regions. 
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Fig. 2. Comparison of kinetic modeling estimated V; and ML predicted V; in the cau- 
date nucleus and in the putamen, for different genetic subgroups. The two Asterisks in 
the plot indicate statistical difference between two groups. 


values within MAB-PD and HAB-PD, which is not the case for caudate V; 
values, estimated using kinetic modeling. Still we have consistent findings for 
HC group in both putamen and caudate V,, and for PD group with regard to 
putamen V, values, which can be considered as satisfactory results. 

In Fig. 3, we conducted a paired t-test to determine if there were any sig- 
nificant differences between the reference and predicted values of V;. We found 
that predicted V, are significantly different from reference V; for MAB-PD and 
MAB-HC, respectively in the caudate nucleus and putamen regions. On the 
other hand, we have a good agreement between ML and kinetic modeling results 
for the remaining groups, in both regions, highlighting the consistency of our 
ML findings. 


34 O. Gassara et al. 


(B) 20 gi a Caudate nucleus putamen 

i © Ref HAB_HC © Ref_HAB_HC 

© Prd HAB HC © Prd HAB_HC 

T V Ref HAB PD V Ref HAB_PD 
N V Prd HAB_PD V Prd HAB_PD 
& O Ref MAB HC © Ref MAB_HC 
x O Prd MAB HC © Prd MAB_HC 
A Ref MAB_PD @ Ref MAB_PD 

A Pdr _MAB_PD A Prd MAB_PD 


Caudate nucleus putamen 


Fig. 3. Paired t-test showing differences of reference V; and predicted V; within subjects 
of the same subgroup, in both caudate nucleus and putamen tissues. The two Asterisks 
in the plot indicate statistical difference between two groups. 


In summary, conventional methods used for quantitative evaluation of PET 
imaging are achieved by means of kinetic modeling, based on compartmental 
and non-compartmental approaches. This requires an accurate measurement of 
IF as well as the application of complex kinetic modeling approaches depending 
on the used tracer. One of the limitations of compartment modeling is that these 
models use iterative fitting including IF to calculate the least squares between 
the measured data and the model data, which can lead to problems of overfit- 
ting and lack of reproducibility. In particular, an inappropriate IF often leads 
to imprecision of the assessed rates. In contrast, our proposed method, based 
on machine learning approaches and the [!8F]-FEPPA radiotracer dataset, pro- 
vides a robust and reproducible solution. It is independent of the input function, 
representing a novelty in the quantitative analysis of PET. 


4 Conclusion 


In this paper, we investigated the feasibility of non-invasively estimating the Vs 
of [!SF]-FEPPA radiotracer, an indicator of neuroinflammation, using its activity 
concentration in brain tissue. We used several non-linear regression models to 
predict the [‘8F|-FEPPA V; in 31 brain regions of interest over 24 patients with 
Parkinson Disease and 20 healthy subjects. The XGBoost model showed the 
best results with a MAE of 2.6. Bland-Altman analysis results indicate that 
predicted V, are in average very close to the reference with a bias of 0.23 F 2.82. 
We also found that significant main effect of genotype on [!SF]-FEPPA in both 
caudate and putamen have been preserved by predicted V, values (p < 0.05) for 
the majority of groups. The results of paired t-test indicate that the difference 
between predicted and reference V; is not statistically significant in 6 out of 8 
groups. This study opens a new research direction in applying machine learning 
algorithms to provide a non-invasive and efficient tool to predict [!8F]-FEPPA V, 
values, a hallmark of neuroinflammation that is believed to be a potential trigger 
for Parkinson’s disease development. As part of future work, our goal will be 
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to develop predictive models to estimate the four pharmacokinetic parameters, 
namely K1, k2, k3, and k4. By accurately deriving these parameters, we aim to 
improve the estimation of the physiological parameters of the total volume of 
distribution V, and the binding potential BP. 
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Abstract. Activities of daily living (ADLs) are basic self-care tasks that are nec- 
essary for independent living. It is also used to assess aging adults’ functional 
ability. As people age, they experience impaired awareness, which affects their 
reasoning, cognitive ability, and consequently impedes their ability to manage 
or supervise ADLs successfully. Impaired awareness has a potential economic 
impact on aging adults. For example, recent studies have shown that aging adults 
with impaired awareness use more energy in homes due to deviation from normal 
execution of ADLs. This often results in incompletely executed ADLs which are 
left unattended or unsupervised. Unsupervised ADLs involving appliances with 
high wattage ratings (e.g., TV, air conditioner) can lead to an increase in the cost 
of home energy services used by aging adults who desire to age-in-place. Thus, in 
this paper, we propose a situation-aware framework that leverages a smart home 
resident’s context information with respect to unattended ADLs in mitigating 
increased energy consumptions in smart home environments. 


Keywords: Smart home - machine learning - ADLs - internet of things (IoT) - 
aging-in-place - situation awareness - aging adult 


1 Introduction 


A smart home environment is an intelligent and inclusive residential space designed to 
enable aging adults to live comfortably and independently [1]. It is retrofitted with a 
network of sensors and IoT devices that can detect events, monitor, and recognize the 
activities of daily living (ADLs) of residents, respond, and adapt to their needs [2-4]. 
For example, smart homes can trigger a warning to notify residents of a potential fire 
hazard if they forget to turn off the stove. ADLs, such as using technology, prepar- 
ing meals, and sleeping, are important tools used to assess an aging adult’s functional 
ability [5]. For instance, if aging adults consistently leave ADLs unsupervised (e.g., 
resident frequently forgets to turn off the TV before going to bed or did not turn off 
the light after using the restroom), it may be an indication of a decline in their health 
and well-being. Several studies have shown a strong correlation between the amount of 
energy consumed in homes and how aging adults perform their ADLs. The projected 
population of aging adults (aged 65 and above) is expected to exceed 70 million by 
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2030 [6], and as they continue to age in place, the amount of home energy consumed 
will increase, resulting in higher costs for home energy services. Aging adults belong 
to an economically vulnerable group with limited options for consistent income [7, 8]. 
Therefore, it becomes crucial to develop a robust framework for energy management in 
older adults’ homes to mitigate increasing energy consumption from unattended ADLs 
and to alleviates the associated financial burden on aging adults who choose to age in 
place. To address this, our study presents a situation-aware framework for mitigation 
of increase in energy consumption from detected unattended/unsupervised ADLs in 
smart home environments. Our proposed framework is designed to help residents sense 
and perceive unattended ADLs in the smart home environment, as well as the states of 
smart home devices/appliances, especially during episodes of impaired awareness. The 
remainder of this paper is organized as follows: Sect. 2 presents the related work, Sect. 3 
highlights the proposed framework and methodology, Sect. 4 discusses the experiment 
design, the results and interpretation are presented in Sect. 5, and in Sect. 6, we present 
the conclusion and future work. Please note that we will use the terms ‘unattended’ and 
‘unsupervised’ interchangeably throughout this paper. 


2 Related Work 


This section presents the vision of this paper and concepts used in this study. 


2.1 Impaired Awareness in Activities of Daily Living 


Activities of daily living (ADLs) are self-care tasks performed to enhance an individual’s 
quality of life (QoL) [9]. ADLs may include basic tasks such as grooming, dressing, 
and toileting, as well as more complex tasks that require stepwise procedures for their 
execution, for example, preparing meals, watching TV, and doing laundry [10]. As people 
age, they are more prone to age-related cognitive decline, which affects their reasoning 
and thinking skills, thereby impeding their ability to successfully manage or supervise 
complex ADLs [11]. When aging adults are unable to supervise ADLs effectively, it may 
suggest that they suffer from impaired awareness [12]. Impaired awareness causes aging 
adults to underestimate their functional decline and can have economic consequences 
and compromise their safety when they choose to age in place [13, 14]. 

For example, an aging adult with mild memory loss may forget to turn off the TV 
and/or the air-conditioner (AC)/fan in the living room before going to bed or may forget 
to turn off the light bulbs after using the restroom or may forget to turn off the stove 
and/or faucet after preparing a meal. Each of these scenarios has economic and safety 
consequences. For instance, if a resident fails to turn off the TV, AC, fan, and light 
bulbs for a long period of time in moments that suggest that the ADLs are incomplete 
or unattended, this could result in an increase in the home energy consumption and 
consequently raises the cost of home energy services. Moreover, when the stove or faucet 
are not turned off and left unsupervised for a long period of time, it may constitute a 
hazard or risk of accidents in homes (e.g., such as fire outbreak or slips due to a wet 
floor. However, the main objective of this study is to mitigate the potential economic 
consequences in terms of energy consumption from unsupervised ADLs. 
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2.2 Situation Awareness for Energy Use Management in Smart Homes 


Situation awareness refers to the ability to perceive or sense devices in the environment 
at a given time instance and location, understand their operational context, and predict 
their future states or status [15]. Smart home environments consist of smart appliances 
and IoT devices that are connected and communicate via a wireless network. These 
smart appliances and IoT devices are tools used for the execution of ADLs. When 
aging adults develop impaired awareness, it impedes their ability to properly supervise 
and manage ADLs, causing them to leave one or more appliances in operating states 
and unattended, which leads to increasing home energy consumption. To address this 
problem and enable aging adults to continue live optimally and independently, a situation- 
aware framework can be integrated into smart home environments. This framework 
enables aging adults to understand and perceive the contextual and operating states 
of the appliances used for the execution of their ADL in smart home environment and 
consequently help to mitigate unintended energy consumption. For example, when smart 
home appliances such as a TV, fan, or AC are in an operational state, each produces a 
distinct sound. These sounds provide useful context data that can be leveraged to gain 
an understanding of an ADL scene. Similarly, passive infrared (PIR) sensors can detect 
motion or human presence within a space at a given time to indicate if an ADL is 
supervised or unsupervised. By leveraging the predictive power of advanced machine 
learning algorithms and applying them to the robust context data from the smart home 
environments, aging adults can regain the awareness required to control the state of home 
appliances when left unattended. This can help prevent unintended energy consumption 
in unsupervised ADLs and alleviate the potential financial burden from increased energy 
use. 
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Fig. 1. Situation-Aware Framework for Energy Savings in smart home 


Situation-Aware Framework for Energy Saving in Unattended Activities 41 
3 Proposed Framework and Methodology 


The objective of our proposed situation-aware framework is to enable aging adults to 
understand and perceive the contextual and operating states of the appliances used for 
the execution of their ADL in smart home environment, and to mitigate unintended 
energy consumption when ADLs are left unsupervised. Specifically, we focused on 
three main smart home appliances that consume more energy in homes: TV, AC, and 
fan. These appliances were used as a case study for our experiment. Additionally, the 
proposed framework includes sensors for sensing lightening from lightbulbs spaces (e.g., 
restroom) and where human presence is not detected after an ADL has been initiated 
but not completed as would be expected for a normal execution of the ADL. In such 
scenarios, the user receives a notification to shut off the lightbulbs. 

Figure | provides a description of the framework and its components. The smart 
devices are connected and controlled through the Alexa Echo Dot. The Raspberry Pi 
serves as the control hub of the smart home environment, detecting the presence of light 
and sound through its connected sensors. This helps to detect the presence of a resident 
within a space. If no human presence is detected in space, it sends a notification to the 
resident’s phone instructing them to turn off any detected devices that are left unattended 
or not turned off. The Alexa Echo Dot can receive requests initiated by the user from our 
custom-built mobile app, allowing them to turn off the smart devices when they receive 
a notification on their phone from the Raspberry Pi. 


3.1 Audio Source Separation 


Blind source separation (BSS) refers to receiving a set of source signals from a mixed 
signal or a set of mixed signals with very little information of the corresponding source 
signals [16]. We require a separator for sound separation in our experiment and propose 
to use BSS to filter out a mixture of sound into its respective audio waveforms. We 
considered four main smart appliances within the smart home set-up. 


IVA Variations Used. We employ four different variations of Independent Vector Anal- 
ysis (IVA) in our study. Two of these variations are based on the gradient-based method. 
NaturalGradLaplaceIVA and GradLaplaceIVA. The other two variations use the aux- 
iliary approach: Auxiliary Laplacian IVA (AuxLaplaceIVA) and Auxiliary Gaussian 
IVA (AuxGaussIVA). In the auxiliary approach, AuxLaplaceIVA utilizes the Lapla- 
cian function as its source model, while AuxGaussIVA uses the Gaussian function. On 
the other hand, both gradient-based methods consider the Laplacian function as their 
source model. However, they differ in how they optimize the KL Loss through gradient 
descent. Specifically, GradLaplaceIVA is like a gradient descent with momentum, as it 
incorporates an additional momentum hyper-parameter along with the learning rate. 


Why Dual Channel? As part of our experiment, we assumed the presence of two 
functioning appliances at a given time. This assumption is based on the observation that 
the AC, fan, and heater are typically used independently without overlapping. When the 
room temperature is warm, residents are more likely to turn on the AC and at a higher 
temperature rather than putting on both the fan and the AC at the same time. Similarly, 
the resident would likely turn on the heater when the room temperature is cold. The 
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fan is considered in scenarios with moderate temperatures. Additionally, we assumed 
the TV would always be turned on in all three scenarios. To avoid increased time and 
computational complexity, we tested the separator using three channels. However, since 
this significantly increases the processing time, we opted for a dual-channel method. 
When only one device is functioning in the dual-channel scenario, we modified the 
system to separate it as a blank output. For instance, an output prediction may include 
‘TV’ and ‘blank’ or ‘fan’ and ‘blank’. 


Why Noise? An important assumption when using IVA is that the input sources should 
not be correlated with each other. However, in the case of a mixture such as the TV and 
fan, the [VA model would encounter an error due to their high correlation. Despite having 
different frequencies and amplitudes, the rate of change of the component is equivalent 
for both sources. To address this, we introduce noise to one of the source samples. There 
are several important parameters in the IVA function. These include the basic machine 
learning parameters such as the type of loss, number of epochs, and number of iterations. 
Additionally, two significant parameters are the algorithm spatial and apply projection 
back. The algorithm spatial parameter can take the values ‘IP’ or ‘ISS’, which determine 
how the algorithm iterates its demix filter. The second parameter, apply projection back, 
is a scaling function applied to the source and mixture to prevent larger values and reduce 
complexity. It can be set to either True or False, depending on whether we want to apply 
it or not. 


Hyper-parameters. The essential hyper-parameter for the model is the noise co- 
efficient (n) which refers to the degree of noise added to the input source. This is 
done to prevent the sources from being highly correlated to each other. The other 
hyper-parameters include the learning rate and the threshold value, which are com- 
mon hyper-parameters while using machine learning models. The learning rate is used 
for optimization while the threshold value stops iterations once loss is at minimum. 


4 Experiment Design 


4.1 Audio Classification 


Audio data for classification in most scenarios is pre-processed by conversion to mel 
spectrograms or by manipulating data through short time Fourier transform (STFT) 
[17]. The short time Fourier transform is a Fourier-based transform which determines 
the sinusoidal (sin) frequency and content of the local sections of a signal under study 
with respect to time [18]. Once we receive the spectrograms, the research problem is now 
reduced to that of a computer vision problem, hence, we rely on powerful convolutional 
neural network (CNN) architectures for the feature extraction. 


STFT. The audio recordings were downsized to 16 kHz and then transformed from 
waveforms into the time frequency-domain signals by computing the STFT. The STFT 
splits the signal into windows of time and runs a Fourier transform on each window, 
preserving some time information, and returning a 2D tensor that you can run standard 
convolutions on. 
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Spectrogram. In the following step, the STFT (number of samples perseg = 255, 
overlap = 124, nfft = 256) was applied to the waveform signals to obtain the spectro- 
gram images of size 129 x 124 (frequency x time) which was then fed into a simple 
convolutional neural network to train the model. 


CNN Model. The CNN model has a convolution layer to down sample the input to 
enable the model to train faster and a normalization layer to normalize each pixel in 
the image based on its mean and standard deviation. The CNN model consists of four 
weight layers used sequentially as follows: conv2—32, conv2—64, maxpool2, FC-128, 
and FC-4. 


Important Parameters. In our experiment, our parameters with the model include 
an epoch value, which is 10, along with a batch size of 64 which was trained on 712 
audio datasets with a 4-class label: “TV”, “AC” and “Fan”. The loss function used is 
SparseCategoricalCrossEntropy since it is a classification problem. The optimization 
parameter used is Adam’s Optimizer. We use the early stopping parameter to end itera- 
tions once minimum loss is reached per iteration. We also set the train-test split parameter 
as 0.6-0.2-0.2 with 60% for training data, 20% testing, and 20% for validation. 


4.2 DashHome Audio Dataset Collection 


The DashHome dataset consists of 60 s audio recordings of smart devices sound including 
TV, AC, fan and heater. There are 712 samples in.wav file format. The recordings were 
collected in a quiet room with a recording device at a sample rate of 44 kHz and then later 
re-sampled to 16 kHz. The AC and fan sounds were captured at different temperatures 
and fan speed, the TV sound was recorded with music or a show playing at volume 
40, and the heater sound was captured when in high and low temperature mode. We 
consider the heater as the input to extract features from it, however it is not an output 
label, hence it is not considered in our output computations. The separator was trained 
with dual audio mixtures of fan-AC, fan-TV, AC-TV, and TV-heater. Each mixture was 
10 min at the highest sensory level of the digital recorder. The TV sound was playing 
at a volume of 40, with the fan speed at four and AC at high. It should be noted that 
despite a couple of mixtures being trained on the heater, it was not considered for the 
calculation of ground truth device probability since the heater sound is very difficult to 
separate and there are few samples that gives us heater outputs. 
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4.3 Smart Devices Setup 


The smart devices used for this study were Alexa-enabled devices connected to the 
Amazon Echo. They include Sony X90J 55 Inch TV, Vornado 660 AE Fan, Rollicool 
14000 BTU Smart Air Conditioner and Philips Hue Smart A19 Lights. In the Alexa 
app, The IFTTT trigger skill (a virtual button that can be used to perform a request 
at the event of button pressed) was enabled and linked to the Alexa account. Then, an 
Alexa routine was created such that when the trigger button is pressed then the devices 
are instructed to be turned off. Finally, a webhook URL to trigger the button from a 
URL request was generated using an applet on ifttt.com using the logic “If receive a 
web request then Trigger the IFTTT Button’, which will then issue the corresponding 
command to the Alexa device to turn off the smart devices automatically. The Raspberry 
Pi 4 with a configuration of 8GB ram and 64GB storage was used in this study with the 
Raspbian OS loaded into memory. The sensors connected to the Raspberry Pi include 
the microphone for collecting sounds from the environment, photoresistor for detecting 
the sensitivity of light bulbs, motion sensor for user location tracking. The function of 
the Raspberry Pi is to continuously collect the environmental sounds and detect if the 
sound is classified as one of the smart devices. 


5 Results and Interpretation 


5.1 Audio Classification 


We use a custom-made CNN model for our classification, and we achieved an accuracy 
of 84%. Further, we considered a threshold value of 0.30. If our probability is greater 
than the BSS threshold, we consider both outputs for the mixture. 


5.2 Audio Separation and Classification 


Initial Studies. We considered the following separator models which were trained on the 
respective source appliances. ‘Ag’ represents Auxiliary Gaussian, while ‘Al’ represents 
Auxiliary Laplacian. We considered only the auxiliary models as opposed to the gradient- 
based ones since they have a much better performance comparatively. ‘ip’ and ‘iss’ are the 
demix filter iteration differences, and the number at the end denotes how many iterations 
the model is trained for. We considered an effective mix of the different iterations, 
models, and methodologies to derive our output. 


1) Ag_ip_TVAC_200. 
2) Ag ip TVFAN_200. 3) Ag ip TVHEATER_200. 4) Ag iss FANAC_300. 
5) Ag iss FANAC_200. 6) Ag iss FANAC_100. 7) Ag iss TVAC 300. 8) 
Al_iss_FANAC_200. 9) Al_ip_TVAC_200. 10) Ag_ip_TVFAN_200. 
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Table 1. The probability of each ground truth smart device calculated from the BSSP 


Ground Truth Device 


Classification Output 


Separation - 


Calculated Probability 


above threshold (0.30) | Classification Outputs | (up to three decimals) 
AC Fan & AC Fan x 13, AC, Fab x 8 | 0.045 
Fan Only Fan Fan x 22 1.00 
TV Only TV TV x22 1.00 
Fan + AC Fan & AC AC, Fan, AC, Fan, 1.00 
AC, Fan, Fan x 2, AC, 
Fan, Fan, AC, AC, 
Fan, AC, Fan, AC, 
AC, AC Fan, AC, Fan 
TV + Fan TV & Fan TV x 6, Fan x 4, TV, | 0.77 
Fan, TV, Fan, TV x 8 
TV + AC Fan & TV Fan, TV, Fan, TV, Fan, | 0.45 
TV, Fan x 4, Heater, 
TV x 3, Fan, TV, Fan, 
TV, Fan, TV, Fan, TV 
Fan + AC + TV Fan & AC TV, Fan, TV, Fan, TV, | 0.69 


Fan x 8, AC x 2, Fan, 
AC, Fan, AC, Fan, 
AC, Fan 


About Table 1. 


The first column represents the input source. 
The second column represents the output given by just the Classifier (experiment 1). 
The Classifier gives a probabilistic output; the sources having a probability above the 
threshold value 0.30, are considered. 


The third column is the output that we get when we pass through 11 different sepa- 


ration models that are coupled together following which we classify each output that 
we receive from the separator (experiment 2). 


The fourth column gives the probability calculated by a probabilistic formula that 


we have developed in the paper, the BSSP (blind source separation probabilistic) 


formula. 


46 R. O. Oyeleke et al. 


The BSSP Formula. 
[N 
5 min T (1) 
= N 
1 


where, 


e iis a particular input source summed up to the total number of input sources involved 
(E.g., Fan, TV, and AC). 

e N is the total output samples received from the coupled separator models; 11 models 
from each dual-channel model (11*2 = 22 total outputs). 

e xis the number of times a predicted output source occurs for each input. 


Model Comparisons. Based on Table 1, we can conclude and compare the models 
as follows: 


e The Ag_iss_TVAC is the only model that was able to detect the AC as AC, hence it 
performed better in all scenarios. Training it over the AC dataset makes the difference 
that gives us the AC as output instead of the Fan. 

e Models trained on TV and fan will never give an AC output, irrespective of the 
iterations and model. 

e Ag _iss_fanac_300 and Ag_iss_FANAC 200 give the same output therefore it can be 
inferred that an auxiliary-based model is independent of the iterations. 

e Ascompared to Laplacian, the Gaussian model has a much-refined output and higher 
probability in general. 


c . e wa ' d 


Fig. 2. Loss curves for the separator model. 


Understanding the Loss Curves. The following points explain the loss curves 
represented in Fig. 2: 
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e (a) and (d) are gradient-based approaches, and we can see that in the output plot. 
There is a decrease in loss with each iteration. 

e (b) and (c) are auxiliary approaches which showed a sharp decrease in loss hence the 
output is independent of iterations. 

e Comparing (a) and (c), we see that the natural gradient Laplace (ngl) algorithm 
minimizes towards a lower loss value than the basic gradient algorithm (gl). Also, 
ngl reaches the minimum loss at a lower iteration value as compared to gl 

e On the other hand, on comparing (b) and (d), the gaussian auxiliary (ag) method 
minimizes the loss to a lower level as compared to the Laplacian, therefore being the 
best model out of all 4. 


Table 2. Output of each machine learning model. 


Ground Truth | Classthication | Audio Seperation + Clisicatn output 


UIETVAG | AUIE IVAN | AU IP IVE ISA FANAC I | AU DS FANAG CI | AU S ANAC | AGLI | DY TANAC | AL IP | 
AC FAN FANFAN | FANFAN | FANFAN FANFAN FANFAN FANFAN J FANAC  JEANFAN J FANFAN | FANFAN FANFAN 
FAN FAN FANFAN | FANFAN | FANFAN FANFAN FANFAN FANFAN ANAC) | FANFAN J FANFAN | FANFAN FANFAN 


V V NV NN WN WV WIV NIV WIV WN NVO NN WIV 
FANAC | FAN ACFAN | ACFAN | ACHAN ACHAN ACEN FANAC ACHAN | ACFAN ACAC | FANAC FANFAN 
WAAN | TV WV | IV WIV FANFAN FANFAN TVEAN WEAN NN NNV INN WN 
TWAC | FAN PANTY} FANT JPNT FANFAN FANFAN HEATER TV} ACTV FANTV FANTV FANTV FANTV 
FANACTV | FAN TEAN | TVFAN TVEAN FANFAN FANFAN FANN =} ANAC] ACFAN ACHAN | ACAN ACAN 


5.3 Energy Savings Outcome 


Suppose an older adult resident turns on the TV, AC, and fan in the living room at 7:00 
PM and then goes to bed at 8:00 PM and forgets to turn off the devices in the living 
room. When the resident wakes up the next day, the energy consumed during that period 
would have incurred a cost with zero service value because the resident was not present 
in the living room area where the ADLs were left unattended. 

However, the Raspberry Pi component of our proposed framework can periodically 
detect human presence, listen to the sound emitted by the appliances, and infer its 
source as TV, fan, AC, or heater. This ADL context information is used to determine 
if notification needs to be sent to the resident’s mobile phone instructing them about 
appliances that are left unattended and are to be turned off to save energy consumption 
and reduce cost. To provide a more specific estimate of energy savings, we can calculate 
the energy consumed every 30 min or hour based on the power consumption as shown in 
Table 2. The assumptions to consider during these calculations are as follows: (Table 3) 


e Weassume the resident turns off the appliance as soon as they receive the notifications. 

e Ittakes the model 30 s to a minute to figure out which appliance is working. Therefore, 
we neglect this short span and consider a time of half an hour as well as an hour, 
to calculate the energy consumed if the resident had left the appliances on for that 
period. 


48 R. O. Oyeleke et al. 


Table 3. Energy savings from smart devices in unattended ADLs. 


Ground truth Power rating of each Energy saved in Energy saved in 

(Smart devices) device/combination of | unattended ADL fora | unattended ADL for a 
devices (Watt) duration of 1 h duration of 30 min 

(W-hour) (W-hour) 

AC 1470 1466 733 

Fan 54.7 50.7 25.35 

TV 219 215 107.5 

Fan + AC 1524.7 1520.7 760.35 

TV + Fan 273.7 269.7 134.85 

TV + AC 1689 1685 84.25 

Fan + AC + TV 1743.7 1739.7 869.85 


e The power rating of the Raspberry Pi is four watts, which shall be considered in our 
experiment while calculating the energy spent to run it. This brings us to our worst- 
case scenario: If the resident has not used any appliance for a day, but has left the 
Raspberry Pi functioning, it would lead to energy being wasted. 

e We calculate the energy saved in kWh which is given by the formula: 


E= Pa k Ty —Pm*T (2) 


where, E is the energy saved for time T,, Pa is the power rating of the appliance, Pm 
is the power rating of the energy management module (Raspberry Pi), 7; is the time of 
study. 


6 Conclusion and Future Work 


The aging adult population belongs to the low-income category of our population and is 
susceptible to age-related impaired awareness. Impaired awareness limits aging adults’ 
ability to successfully supervise ADLs and results in unanticipated economic conse- 
quences. For example, when home appliances are in operational states and left unat- 
tended, it leads to an increase in the amount of home energy consumed and consequently 
raises the cost expended on home energy service. It is therefore imperative to develop an 
effective solution to enable aging adults with impaired awareness to mitigate unintended 
energy use from unattended ADLs so that they can continue to afford to age-in-place. In 
this study, we proposed a situation-aware framework for mitigation of unintended energy 
consumption in unattended ADLs. The framework includes low-cost sensors and IoT 
devices such as Raspberry Pi, sound, light and motion sensors. In addition, the frame- 
work incorporates machine learning models that uses robust context data generated from 
the smart home appliances to enable the user to understand their ADLs environment, 
perceive smart home appliances and their operating states to mitigate unintended energy 
consumption when ADLs are unattended. The results obtained from the experiments are 
promising. For our future work, we plan to test the efficacy of our proposed framework 
in senior retirement homes in using longitudinal study. 
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Abstract. Dashboards are powerful electronic tools that can provide action- 
able insights for healthcare professionals, especially in support of the increas- 
ing senior population. With advancements in technology and IoT infrastructure, 
remote patient monitoring has become a feasible option for healthcare profes- 
sionals through dashboards. To best serve the diverse needs of healthcare pro- 
fessionals, dashboards should be tailored for each user, considering their roles, 
interests, and priorities. In this study we proposed AMI-Dash, a solution allows 
for dynamic design and information visualization to address the diversity in needs 
and priorities among different dashboard users while maintaining a high-level of 
performance, as evaluated through several technical aspects. 


Keywords: Dashboards - Internet of Things (IoT) - Remote Health Monitoring 
(RHM) - Healthy Aging - Senior residences - Deployment 


1 Introduction 


Dashboards can be a useful tool to deliver care to older adults while providing real- 
time health information to healthcare professionals. The advancement in the Internet of 
Things (IoT) and sensor technology allows for constant monitoring of seniors’ health 
conditions, which is beneficial in reducing the work burden of healthcare professionals. 
This is especially important when we consider the shortage of healthcare professionals 
and the swelling population of older adults, leading to more pressure on the healthcare 
system and inevitable need for supportive solutions. 

The population of older adults is increasing in numerous developed countries, includ- 
ing in Canada, where nearly a third of its population belongs to the baby boom genera- 
tion [1]. This is putting more pressure on the already stretched healthcare system as the 
demand for healthcare and life aid services for baby boomers will continue to increase[2]. 
Remote Health Monitoring (RHM) is a new modern approach that has the potential to 
help healthcare professionals in Senior Residences by continuously monitoring individu- 
als’ health factors and transmitting them to healthcare professionals [3]. This can reduce 
the burden on healthcare professionals and the pressure on the healthcare system, as well 
as reduce costs and improve the quality of life for older adults. RHM systems generate 
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heavy streams of data, but it is important to deliver this information to healthcare pro- 
fessionals in an understandable manner. Dashboards are powerful interfaces that can use 
RHM and act as a medium to deliver meaningful information to healthcare professionals 
and enable them to make informed and timely decisions. However, there are several prob- 
lems associated with dashboard development and deployment in healthcare, including 
limited capabilities to respond to diverse needs, preferences, and roles in healthcare, the 
complexity of using multiple independent dashboards, and the high cost of dashboard 
development [4, 5]. The design of dashboards must constantly adapt to changing needs 
and be tailored to the specific preferences and roles of healthcare professionals. The 
large number of dashboards can lead to islands of knowledge and make it difficult for 
healthcare professionals to gain insights they need. Additionally, existing development 
approaches make it difficult to expand solutions to other healthcare centers, increasing 
costs. Therefore, the development of dashboards requires a deep understanding of real 
needs of health professionals and the healthcare workflow. 

We aim in this study to achieve a tailored solution for a dashboard in the context 
of healthy aging and senior residences. We are interested in reducing the information 
load and mental workload of healthcare professionals and improving their effectiveness 
by providing them with tailored dashboards. Therefore, the overall aim of this research 
is to provide a cost-effective and efficient monitoring mechanism to reduce the work- 
load of healthcare professionals. More specifically, the presented study aims to provide 
personalized dashboards integrated with the AMI-Lab platform [6], a complete IoT 
infrastructure for dynamic and quick deployment of IoT systems which is already being 
used in several research projects to follow up remotely older adult status. 

The rest of the paper is organized as follows: Sect. 2 introduces the diverse approaches 
to handling tailored dashboards. Section 3 presents the proposed approach, the imple- 
mentation as well as the results of the experiments and evaluations conducted. We finally 
outline the conclusion in Sect. 4. 


2 Literature Review 


There are several studies on building tailored dashboards with diverse approaches to 
manage the requirements of each user and how to handle tailored dashboards accordingly. 
We can categorize the key strategies into three: a) Facilitate source code generation, b) 
Connecting to existing BI tools and c) Personalized solutions. 


a) Facilitate Source Code Generation: Achieving tailored solutions in code level was 
one of the approaches taken in the literature. A number of studies took software pro- 
duction line in the literature [7]. This approach uses code templates to achieve dynamic 
creation of dashboards. User requirements and usage context is fed to the generation 
engine to generate the source code of the intended dashboard. Similarly, A model-driven 
approach developed by Palpanas et al. [8] follows the same approach using defined mod- 
els. The difference in this study is that information provided as input for their system is 
fine-grained based on dashboard structure and it asks to provide more details including 
navigation, access controls and templates. Based on this provided information, a com- 
plete functional code for the dashboard is generated. In all mentioned works that use 
code approach, the developers have access to final source code after generation of the 
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dashboard and they can modify it to meet their desired customization. Therefore, this 
solution provides a high level of flexibility. On the other hand, this approach depends on 
the existence of developers and technical people, and it cannot be used by final users of 
the dashboard. Moreover, existence of source code means that the final solution should 
pass all steps required for deployment of a software, ranging from compiling the code 
to deploying it on the final production system. Nevertheless, none of the studies using 
this approach mentioned provided specific plans for deployment which can be a very 
resource consuming task if it is not performed properly. 


b) Connecting to Existing BI Tools: Interestingly, a number of existing solutions 
used external BI tools for the final rendering of the dashboard. Tundor et al. [9] pro- 
posed declaration description of the dashboard, and the final output of their system is a 
configuration layout which can be used by other BI solutions to represent the designed 
dashboard. Similarly, Santos et al. [10] proposed a knowledge graph which uses an 
ontology approach to generate dashboards automatically for smart cities. They used API 
methods to communicate with other BI solutions and reduce the efforts required to create 
the dashboard. Although leaving the responsibility of rendering the final dashboard can 
dramatically reduce development time and cost, it has a number of clear drawbacks. The 
main drawback of this approach is that the software cannot work standalone, and the 
user needs to link the output of proposed solutions to another external tool to be able to 
visualize data. Moreover, not all BI tools support common forms of visualization. An 
element on the designed dashboard might not be able to render on certain BI tools and 
therefore they cannot be shown on the final dashboard. 


c) Personalized Solutions: A number of studies aimed to personalize the dashboard to 
users based on their need. We witnessed two different perspectives to achieve this goal. 
One perspective was to perform this procedure automatically based on the behavior and 
usage of end users while they are using the dashboard. For instance, Belo etal. [11] aimed 
to restructure the dashboard elements based on information collected during end-user 
usage. The other perspective requires explicit definition of user needs and then tailoring 
dashboard based on declared needs. This idea can be seen in studies of Ines et al. [12] 
and Tundo et al. [9]. More specifically, Ines et al. [12] designed a questionnaire to extract 
the needs of the end user and use collected information later to provide personalized 
dashboard. Similarly, Tundo et al. [9] also need an explicit declaration of intended 
dashboard. In both perspectives, authors assumed that the dashboard has the required 
visualization elements to reply to the needs of users and the main goal is to make existing 
elements more accessible for users. They did not mention how they want to handle the 
changes in users’ needs over time or how to provide different users with different data. 


Overall, the existing solutions in literature provided limited flexibility in terms of 
design and supporting new visual elements, most of the proposed solution focused on 
linking existing visual element to appropriate data sources and create a dashboard by 
combining these elements. Similarly, the majority of proposed solutions have not consid- 
ered the need for adjusting dashboard after its creation and once the dashboard is created, 
there is no room for customization or trying different possible dashboard designs. In the 
existing solutions, we did not witness the option to provide a flexibility in designing the 
whole frame of a dashboard. Although the content of the dashboard is customized in 
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these solutions, the structure and the frame of the dashboard is fixed for all users and 
use cases, and the need to change the structure is not supported. 

After exploring the literature of tailored dashboards, in the next section we present 
our designed approach. 


3 Design Approach 


To provide a solution to respond to the identified needs and enable healthcare profes- 
sionals to easily create dashboards adapted to each deployment, AMI-Dash has been 
developed in collaboration with other teams in AMI-Lab with agile methodology. This 
methodology enabled us to assess the system throughout the development phase in sev- 
eral iterations and assure integrity of our solution with other elements of AMI-Platform 
to ensure that all parts of the system are working together. Dashboard Studio/Designer 
is the core of our proposed solution which enables healthcare professionals to design 
their intended dashboards and automatically adapt them to different environments based 
on information received from a Configuration Tool. 

We followed a four-step agile approach to design AMI-Dash adaptive dashboards. 
In the 1* step, we defined the requirements for the dashboards by analyzing existing 
work and grouping the requirements into end-user and technical needs. In the 2" step, 
we proposed a solution based on literature and technology to address the identified 
requirements, focusing on the concepts of “dynamicity” and “scalability.” In the 3" 
step, we implemented a web-based prototype called AMI-Dash, designed to be user- 
friendly and adaptable to the real-world environment. In the 4" step, we evaluated the 
prototype through a two-part evaluation process, measuring the technical performance 
against the defined requirements. 


3.1 1% Step: Requirements 


We have identified five categories of technical requirements that involves dashboards in 
senior residences: 


Reg. A) Privacy Prevention and Security: Protected Health Information (PHI) is 
an important factor in healthcare dashboards. Legislation in many countries including 
Canada requires least privilege or “minimum necessary” access to personal health care 
information [13]. In another word, each health provider should be limited to access only 
portions of information that is essential for them to perform their tasks. However, the 
complexity of this requirement lies on the fact that health information is usually stored 
in a central storage for integrity and accessibility reasons [14]. 


Req. B) Fast Easy Deployment of a Dashboard: Deployment refers to delivering the 
final version of the designed dashboard to real users. Nowadays more complex software 
solutions are introduced in numerous domains including healthcare dashboards. These 
modern solutions usually involve adding third party systems to the solution including 
databases, firewalls and caching systems [15]. When it comes to deployment of new 
solutions, all these aspects should be considered to setup which leads to time-consuming 
procedure and needs of technical individuals and paper works regarding administrative 
permission. 
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Req. C) Real-Time Data Visualization if Needed: Real-time data refers to data which 
collected and processed in near real time and is available immediately after it is collected 
by sensing devices. This form of data can be specifically beneficial to reduce intervention 
time and improve patient safety, two important use-cases of dashboards in healthcare 
and senior residences. In both cases, intervention and safety, it is critical for healthcare 
professionals to notify any abnormal condition of the patient or the monitored person as 
quickly as possible. 


Req. D) Low Resource Consumption: Dashboards usually include multiple charts and 
figures along with other visualization elements like maps and images which potentially 
can consume a high amount of resources. With the emergence of new RHM devices 
and IoT sensors, measured data for a large variety of health and environmental factors 
are available. Having such number of measured factors can be interpreted to have an 
even larger number of visualization elements on desired dashboards, which require 
handling a large amount of data and processing. Numerous dashboard solutions [16-18] 
are deployed on devices like tablets and smart TVs with limited computational resources. 
Therefore, it is critical for dashboard solutions to consume low resources of the system 
in order to be able to handle a large number of visualization elements and to be hosted 
on a variety of devices, especially when there is a need to load real-time data. 


Req. E) Scalability: Dashboard solution should be able to handle unpredictable 
requirements in the future. Healthcare domain is very dynamic, and a wide variety 
of needs exists in this realm. With constant advancements in sensors and healthcare 
devices, it is highly likely to have new types of devices available in the market. These 
new devices might need new forms of visualization or new algorithms to enable health- 
care professionals to analyze and interpret the data. Therefore, it is essential to be able 
to support emerging needs in healthcare domain. 


3.2 2"4d Step: Proposed Approach 


Our approach is based on a number of subsystems to form the complete solution (Fig. 1). 
The main part of our solution is the Page-Designer which is responsible for enabling 
healthcare professionals to create and design dashboard pages based on their needs 
and preferences. This designer is equipped with functionality to communicate with IoT 
infrastructure of AMI-Lab, and visual elements of this page designer has attributes to con- 
nect to real devices in IoT infrastructure. As our proposed solution need to integrate the 
dashboard with IoT systems (fast dashboard creation), we need to have semantic infor- 
mation about the deployed sensors, which is received from Environmental Configuration 
Tool (AMI-ECT). 

Once we can manipulate information on the deployed sensors and the deployment 
environment, there is a need for manipulating the representation (view) of this infor- 
mation. Chart-Designer provides healthcare professionals with the ability to manipulate 
dashboard charts with a great level of flexibility. It provides the ability to manipulate 
easily not only the layout of the pages and the arrangement of elements, but also, add 
new components and new visualization types that transform raw data to meaningful 
information. Lastly, Dashboard-Viewer is responsible for rendering the final layout of 
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the designed dashboard for healthcare professionals with all required visualizations. 
This viewer along with previously mentioned parts of the solution are integrated with 
Authentication system of AMI-Platform to separate the design space of each healthcare 
professional and provide user tailored services. 


AMI-Platform 


Environmental 
AMI-Dash Configuration Authentication 
Tool 


IoT 
Infrastructure 


Dashboard 
Generation Chart Designer 
Engine 


Dashboard Dashboard 
Page Designer Viewer 


Fig. 1. Overview of proposed solution 


Following we discuss how our proposed solution replies to the identified require- 
ments: 


Req. A) Privacy Prevention and Security: We integrated an authentication system in 
our solution which handles all the functionality to define users (healthcare professionals) 
and assigning their roles. Based on healthcare professional information, dashboard sys- 
tem applies limitations according to their roles and gives them appropriate data access 
and management permission. As an example, the designer may wish to limit a page to 
be visible only for healthcare professionals with “Nurse” role. Dashboard Studio can 
handle that based on their identifier as the logic of showing a page is handled there. 


Req. B) Fast Easy Deployment of a Dashboard: We proposed a subsystem called 
Dashboard-Viewer in our solution to automate the deployment procedure. Dashboard- 
Viewer is responsible for reading the configuration of requested dashboard from the 
data source. The input of Dashboard-Viewer is a JSON object that contains information 
about the elements inside the dashboard and their arrangement inside the final layout of 
rendered dashboard. This JSON object starts with a ROOT child which lists all its children 
and therefore enables the render engine to detect how to start rendering elements. Once 
configuration is loaded, a render engine in Dashboard-Viewer will parse the configuration 
to extract items and their arrangement in the dashboard and start rendering the designed 
dashboard and load its content. As our Dashboard-Viewer is always running on the 
server, there is no need to configure any additional system and a dashboard is instantly 
available once its design is finished. 


Req. C) Real-Time Data Visualization if Needed: To support visualization of real- 
time data, dashboard components can receive live updates from their data sources and 
visualize data in a real-time manner. For each component there is a long-running back- 
ground task which is responsible for regularly updating data of the component on regular 
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intervals. We also enabled healthcare professionals to adjust update intervals based on 
the sensitivity of the element’s data. For instance, they might set update intervals for a 
heartbeat chart to half a second while setting the update interval of sleep status to five 
seconds since it has less sensitive data. 


Req. D) Low Resource Consumption: Dashboard-Viewer is the core part of the sys- 
tem responsible for rendering layout and contents of the dashboard, making it the main 
target for our efforts to make it efficient. One of our initiatives is to limit refreshing dash- 
board charts to the time that they are visible, this approach can greatly reduce resource 
consumption. If a chart is not visible on the page, it will stop sending any request until it 
becomes visible again. Using Async load methods is another optimization we added. All 
data elements inside the dashboard send their requests for data in asynchronous mode, 
which means that loading the data of each chart is not part of loading the dashboard itself. 
We also enabled the compression on our web server to reduce the size of information 
received from the server, which minimizes the size of data exchange. 


Req. E) Scalability: All the elements inside the dashboard are components, and each 
component can be programed to perform a specific task or visualize information in a 
certain way. Being components, plays a key role in our implementation as it enables the 
system to accept new components so that in case we need a new visualization method 
or dashboard element, we can simply develop and add these components to our system. 


3.3 3'4 Implementation of AMI-Dash 


AMI-Dash is composed of the following three important components (Environment 
Configuration Tool, Chart Designer and Dashboard Designer). 


a) Environment Configuration Tool: We built the Environmental Configuration Tool 
(AMI-ECT) to generate the semantic information about the deployment place precisely 
so that it can be consumed by the dashboard studio to drive the generation engine 
adaptively. AMI-ECT provides a 3D designer in which healthcare professional is able to 
drag-and-drop physical elements that exist in the deployment place to create a floor plan 
of a building, as well as place sensors on it and specify their types (Fig. 2). The output of 
this component is an “Environment Model” which precisely describes the deployment 
area and contains all data needed to adapt the dashboard design. 


b) Chart Designer execution can be seen in the Fig. 3. At the very first step the 
healthcare professional is supposed to create a new chart and specify the title (name) 
of this new chart and declare if it is a static chart or a template chart. If they choose 
template charts, the design can be automatically adapted to new environments provided 
by AMI-ECT. Once this initial information is provided by the healthcare professional, 
the next step is to specify how to fetch data from data sources and define its filters and 
timeframe that should be captured from the source. The easiest option is to ask the end 
user to write a query string. However, there are several clear drawbacks to that approach. 
First, the healthcare professional needs to have technical knowledge about the database 
and its structure and the knowledge to write appropriate queries. Apart from that, even 
if they had the knowledge working which queries, by writing a raw query we limit our 
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logic to a specific database type used in our implementation and if there was any change 
in the database or its structure, it would directly affect the way end-user will need to 
define the query. Therefore, we equipped our system with a number of data panels that 
can help healthcare professionals to define query and filters. 

Animportant aspect of the chart designer is the ability to design templates. Healthcare 
professionals can define charts as a static chart that will be shown as they are designed 
initially, or they can use templates which will be adapted to different spaces declared by 
AMI-ECT. For example, in one deployment we might have a hundred patients equipped 
with an identical sensor (e.g., blood pressure). In this situation, the design of the chart is 
the same in all these cases and the only change is refining the data source of the chart. We 
tackled this problem by invoking a template and linking the blood pressure visualization 
of all these patients to one template. 


Fig. 2. Example of running configuration tool. 
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filters to methods to method and Save chart with 
extract data summarize map fetch dashboard 
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Fig. 3. Overview of all steps in designing charts with AMI-Dash 
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C) Dashboard Designer: The cornerstone of our solution lies in this part of the system 
as it enables healthcare professionals to design dashboard pages and adapt them to the 
real-world environment based on the description provided through AMI-ECT. Dash- 
board Studio consists of five main components which will be explained in the following 
sections. Figure 4 illustrates the high-level design of AMI-Dash Studio: 


Page and 
Template 
Designer 


Page Navigator geaca non 


AMI-Dash 


Studio 


Dashboard Page Generator 
Viewer i 


Fig. 4. Elements of Dashboard Studio 


AMI-Dash studio (Fig. 4) consists of five main parts: 


e Page and Template Designer: The main element in our solution in which healthcare 
professionals can design their desired Dashboards. 

e Page Generation Engine: This engine is responsible for adapting healthcare profes- 
sional designed dashboards to the arrangement of real-world environment declared 
by AMI-ECT. 

e Dashboard Viewer: This part of the system handles rendering designed dashboard 
for final usage by healthcare professionals. 

e Page Navigator: Enable navigation among dashboard pages 

e Authentication Manager: Handles healthcare professional permission to limit their 
access to certain pages if needed. 


3.4 4th Step Evaluation 


We conducted an evaluation study to understand to which extent the proposed solution 
has replied to technical requirements (identified in 1* step). Following details about the 
study. 


Evaluation Setting: AMI-Platform has been deployed and running for months now 
and in several projects located in Canada. We used our solution in two project settings: 
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e Behavior change detection in apartment deployments: AMI-Lab platform is 
deployed in a number of apartments, in which older adults are equipped with a vari- 
ety of environmental and health sensors which continuously monitor the following 
parameters: Motion, Temperature, Humidity, Iluminance, Sleep factors. 

e Medical intervention monitoring: using two wearable devices: 1) Apple Watches to 
follow up Heartrate, Oxygen level of blood, and Number of steps walked; 2) Dexcom 
Sensors to follow up sugar level of blood. 


Evaluation Metrics: To validate the proposed solution, we targeted the full-wing 
metrics that are representative of the technical requirements: 
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Fig. 5. Snapshot of dashboard page designer 


Req. A) Privacy Prevention and Security: Limiting data access to certain roles was 
an important identified problem. In all deployments we needed to deliver different dash- 
boards to our partners. Our implemented version successfully handled loading informa- 
tion for all users and no issue regarding authentication is reported. To assure confiden- 
tiality in the communication channel, we used a tool called “Charles Proxy” to sniff data 
exchange between our proposed dashboard and its server. Manual check of headers and 
content of sending and received packets, confirms that all exchanged data is encrypted 
using standard TLS methods, which makes content confidential, even if it passed unse- 
cured channels as it uses end-to-end encryption. By taking these two steps, our results 
confirm that our implementation is capable of replying to the identified requirement. 


Req. B) Fast Deployment: The proposed solution is deployed in several real apartment 
with the aim of monitoring older adults at home. Dashboards deployed in these apart- 
ments have different floor plans, however, they mostly include five door sensors, seven 
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multi-sensors which measure a number of parameters together (motion, temperature, 
humidity, and lux). A sleep mat is also part of the project to measure the sleep-related 
parameters. Similarly, in another project, we needed dashboards to visualize measured 
data of the deployed Apple Watches and Dexcoms. Once the installment of physical 
devices was finished, we started to design required charts for these projects in Chart 
Studio, followed by creating desired dashboard using AMI-Dash Studio, we were able 
to simply drag and drop the required visual elements and connect them to the appropri- 
ate data sources. The whole process of designing the required charts and the dashboard 
itself took roughly 20 min for apartment deployments. Finally, once the design step is 
finished, the dashboard is immediately accessible with no need of performing complex 
procedures. It worth mentioning that developing dashboards using ordinary methods can 
take several months to even years. 


Req. D) Low Resource Consumption: CPU was in idle mode for our dashboard, 
suggesting enough efficiency to even run the dashboard in other media like smartphones 
and smart TVs.We run the dashboard with 30 figures and 5 images in the dashboard 
along with normal elements (Texts, Buttons and Containers). As it can be seen in Fig. 6, 
the CPU consumption is very little and 97% of the times the 
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Fig. 6. Overall resource consumption of our Dashboard Studio 


The result of our comparison shows that there is almost no difference between the 
visualization of the designed dashboard and the normal web page. Additionally, our 
dashboard is used for a period of two months by AMI-Lab partners and no complaint 
regarding performance was received. 

In another scenario, we measured memory consumed by the dashboard while our 
dashboard was running and contained a certain number of charts. Each time we added 
a new chart and checked how much memory is consumed by our dashboard. Overall, 
the results show a reasonable memory consumption (around 190 MB with 30 charts). 
The trend of memory consumption was increasing as the number of charts increased. 
However, after adding 12 charts this increase started to be less considerable. This has 
roots in our implementation methods in which when a chart is not visible, our system 
stops sending request and allocating resources for it. 


Req. C) Real-Time Data Visualization: In order to measure the delay in visualization, 
we manually inserted a new set of data to our data storage and check how long it takes 
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Fig. 7. Memory consumption and number of charts on the dashboard 


for the new data to be reflected on the dashboard. To this end, we used software called 
“Postman” which is designed to interact with web interfaces and enable us to directly 
communicate with our backend services and add new data to our data source. Our data 
storage model contains a timestamp field, which indicates the exact time a new document 
is inserted in our system. We used this field along with another function which returns 
exact current time (server time) and we added a temporary label on charts to show the 
difference (delay). We set the refresh time of the chart at 100 ms and observed the 
measured delay and in all cases the delay was less than 100ms, proving the capability 
of our system to show real-time data. 


Req. E) Scalability: As we used our solution in real deployments of AMI-Lab we 
faced few use-cases which were not originally defined in our solution. However, as we 
have used a component-based approach in our design, we are able to easily develop 
new components to respond to any new need. As an example, a colleague needed to 
annotate chart data shown on the dashboard, which even needed interaction with data 
sources to store annotated information. The integration process took less than 30 min 
for integration, and the added component worked smoothly with no functional error. 


4 Conclusion 


The study presented in this paper aims to propose a form of tailored dashboards that can 
allow healthcare professionals to create their own dashboards for senior residences. The 
motivation behind this study is to support healthy aging and help healthcare professionals 
deal with the increasing number of senior residents. The need for this study arose due 
to the limitations of existing technological solutions such as Remote Health Monitoring 
(RHM) and the Internet of Things (IoT) monitoring which do not provide a medium for 
healthcare professionals to understand the data generated by these devices. 

The study followed an agile approach in four steps to reach its final solution, AMI- 
Dash. In the first step, the requirements for adaptive dashboards in senior residences 
were identified. In the second step, the solution was outlined based on the context of 
work, best practices in the literature of tailored dashboards and stacked technological 
solutions in AMI-Lab. In the third step, a prototype was implemented based on the 
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outlined solutions. Finally, in the fourth step, an evaluation was performed to measure 
the usability of the solution for the end-user. 

The real needs of healthcare professionals working in senior residences were 
explored through the literature of healthcare dashboards and the studies that were close 
to the context of work. The analysis of these studies led to the extraction of a number 
of requirements for dashboards in senior residences. The final dashboard designer was 
easy for healthcare professionals to use and provided the option to adapt the dashboard 
to different deployment places!. 

There are some limitations to the study such as the current implementation being 
coupled with the AMI-Platform and the lack of integration with other systems and 
managerial aspects. However, future work can involve integrating the solution with 
other IoT infrastructures and existing management tools in senior residences. 
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Abstract. Advancements on the Internet of Things (IoT) have enabled the devel- 
opment of advanced monitoring systems that can track human behavior and vital 
signs in real-time, which can have a real impact in the way healthcare is provided. 
This paper presents a system implementation to monitor and analyze a subject’s 
behavior changes over time using IoT, with the objective of detecting the impact of 
an inhibitor drug on the subject’s activity levels. In this research we present a case 
study by which we showed it is possible to follow the effect of an anticholinergic 
drug by means of an unobtrusive IoT system. We have monitored the physical 
activity of a subject in his residence for seven consecutive months to study the 
effect of the inhibiting drug doses introduced at three known specific timestamps. 
Following, we compared our detection results for the subject’s physical activ- 
ity change timestamps with the medical staff medication doses timestamps. Our 
results show that we can detect the physical activity change at close timestamps 
compared to those indicated by the medical staff. 


Keywords: Physical Activity Change Detection - Outliers Detection - 
Medication Follow-up 


1 Introduction 


The Internet of Things (IoT) is a rapidly growing network of interconnected devices that 
can share data and communicate with one another, and it has revolutionized a wide range 
of industries, including healthcare. IoT devices and systems can be used to gather and 
analyze data on patients, monitor their health conditions, and provide personalized and 
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timely healthcare [1]. One area where IoT is particularly promising is in the development 
and monitoring of new drugs, such as inhibitor drugs used to slow down or stop specific 
biological processes or activities [2]. Monitoring the behavior and activity of subjects 
under the influence of such drugs is crucial to address two main issues: the drug effect 
itself and the pharmaceutical metrics [3, 4]. 

In this paper, we are presenting a proof of concept based on a case study to show we 
can follow the effect of drug intervention on a subject using IoT. To this end, we have 
monitored a subject in his residence where he was given three timestamped doses of 
anticholinergic drug. We used our IoT framework to monitor his daily physical activity 
inside the residence and analyzing the captured timeseries afterwards. To achieve such 
a task, the physical activity is monitored using motion sensors embedded in the ambient 
environment. Following, we used an outliers’ detection technique applied to the physical 
activity of the subject that relied on a sliding window applied to the entire timeseries 
collected. The mean and standard deviation of the subject’s physical activity was quantity 
within the sliding window. We considered the change in the subject’s physical behavior 
to be defined as outliers from the calculated mean and standard deviation gathered across 
the sliding window. 

This paper is organized as following, Sect. 2 comprises the related work in this 
particular field of research. The methodology is presented in Sect. 3. Results and dis- 
cussion are presented in Sects. 4 and 5, respectively. Finally, the conclusion is presented 
in Sect. 6. 


2 Related Work 


There has been significant interest in IoT research within the healthcare domain, resulting 
in the development of various applications for different diseases, settings, and popula- 
tions. A comprehensive review conducted by Lin et al. [5] surveyed representative and 
active IoT application products and categorized their focuses into three care settings: (i) 
acute disease care, (ii) chronic disease care, and (iii) self-health management. 

In acute disease care, it is crucial to collect patients’ vital signs accurately and in a 
timely manner. For example, Gao et. al. [6] collaborated closely with medical profes- 
sionals to identify the key needs in health emergencies. They designed and implemented 
a system that records patients’ vital signs in real-time using wearable sensors, detects 
anomalies through a vital sign monitoring algorithm, and shares the data with authorized 
personnel via a web portal. 

IoT applications in chronic disease management aim to enable remote health moni- 
toring. For example, Al-Khafajiy et al. [7] presented Wearable Sensors for Smart Health- 
care Monitoring System (SW-SHMS), a smart healthcare monitoring system that collects 
physiological data from wearable devices. In their system, the collected data is trans- 
mitted to a smartphone app via Bluetooth and then to the cloud over the internet. The 
processed data is displayed on a monitoring platform accessible by doctors. Another 
example is IMMED (Indexing MultiMEdia data from wearable sensors for diagnostics 
and treatment of Dementia) was proposed by Mégret et al. [8]. IMMED relies on wear- 
able cameras to capture video and audio data of older adults’ daily activities to assess 
cognitive decline. 
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As the importance of health monitoring becomes increasingly recognized, more IT 
applications are being developed to assist with self-health management. For example, 
Sadek et al. [9] proposed an unobtrusive Apnea detection using a bed sensor and Sadek 
et al. [10] proposed a sleep monitoring system using a smart bed. Additionally, smart- 
watches have been utilized in activity recognition [11] and as a stress detection using 
heart rate variability [12]. Another technique for monitoring the indoor daily activity was 
proposed by Ahmed et al. [13] who relied on a thermal sensor array-based (TSA-based) 
system that can monitor sleeping activity, daily activity, and the no-activity class of a 
person. 

Drug intervention management is important for health management and started to 
attract attention of clinical research. For example, clinical researchers are interested in 
evaluating how deprescribing anticholinergic and sedative medications impact people 
behavior. Based on a literature review, Reeve et al. [14] defined deprescribing as “the 
process of withdrawal of an inappropriate medication, supervised by a health care pro- 
fessional with the goal of managing polypharmacy and improving outcomes.” Although 
this definition has not yet been validated, it can be used as a basis for any deprescribing 
drug, even if further work is needed to get expert consensus to develop an internationally 
accepted definition [14]. 

Considering Reeves et al. [14] definition, in recent years there are accumulating evi- 
dence showing health improvement associated with deprescribing anticholinergic and 
sedative in older adults. For example, Ailabouni N. et al. [15] found that reducing anti- 
cholinergic and sedative medications in frail older people was feasible and beneficial 
and resulted in better health outcomes, fewer adverse effects, and improved mood and 
vigor. They also concluded that deprescribing with a patient-centered approach can be 
safe and effective. Dearing et al. [16] evaluated the effectiveness of a pharmacist-led 
medication optimization initiative using an electronic tool called the Drug Burden Index 
(BDI) Calculator in four hospital sites in Canada. This study aimed to achieve three 
objectives: First, to determine the impact of integrating an electronic clinical decision 
support tool, the DBI Calculator, into pharmacist medication optimization activities on 
medication changes and clinical outcomes; Second, to assess whether the effect of the 
intervention varies based on participant characteristics such as frailty status and sex, as 
well as the setting of the intervention, including different ward and hospital characteris- 
tics; and Third, to explore the implementation of the DBI Calculator into pharmacist-led 
medication optimization activities during inpatient admissions. The study found signifi- 
cant changes in DBI scores and clinical outcomes, indicating the potential benefit of such 
interventions. Similarly, Arvisais et al. [17] conducted a study to assess the efficacy of 
a pharmacist-physician intervention model in reducing the use of high-risk medications 
in older adult patients. The study reported that the intervention model was effective in 
reducing the use of high-risk medications. Also, Cossette et al. [18, 19] investigated the 
impact of a computerized alert system-based pharmacist-physician intervention model 
on Potentially Inappropriate Medications (PIM) use. The interventions resulted in a 
higher number of drug cessation and dosage reductions for targeted PIMs compared to 
a control group. In a pilot study conducted by Cossette et al. [20] in a primary care 
setting, an interdisciplinary pharmacist-physician intervention model based on alerts 
generated by a computerized alert system was implemented to reduce the use of PIM 


68 H. M. Ahmed et al. 


in community-dwelling older adults. The study found that a majority of the patients 
included in the study had medication change suggested by the pharmacist, indicating the 
potential effectiveness of this intervention model. These results can be generalized older 
adults living in different regions of the world. Studies conducted in New Zealand [21], 
Taiwan [22], and Texas [23] have highlighted the prevalence of PIM use and associated 
factors and suggested the need for screening tools such as the Beers criteria to identify 
PIMs and interventions to optimize or minimize their use. 

Additionally, the use of PIMs can increase the likelihood of adverse drug reactions 
(ADRs), the harmful or unintended reactions to medications. The prevalence of ADRs 
is yet another significant concern for health care. Specifically, Beijer and de Blaey [24] 
found that among the older adult population one in six patients are for ADR as opposed 
to ~ 4% in the younger population [24]. Various studies have highlighted the burden 
of ADRs in hospital inpatients at large [25]. These studies have identified risk factors 
such as increasing age, admission to medical wards, and the number of regular medica- 
tions. Strategies to minimize the burden of ADRs include computerized prescribing and 
monitoring systems, pharmacist involvement in ward rounds, improved monitoring, and 
education on prescribing practices. Harmful events resulting from PIMs also includes 
adverse drug events (ADEs) [26] that are more likely to occur in older adults with mul- 
tiple chronic conditions as well as older adults taking specific medication classes. This 
highlights the importance of considering the overall health and medication profile of 
older adults to minimize the risk of ADEs. A study by Bressler et al. [27] emphasized 
the need for physicians to be continuously aware of potential adverse drug interac- 
tions in older patients and to consult pharmacists and/or patients to ensure appropriate 
medication prescribing. 

As highlighted by numerous studies [18, 17, 27], ensuring appropriate medication 
prescription and monitoring of older patients is crucial to avoid ADEs and ADRs. This 
emphasizes the need for continuous monitoring of patients’ behavior and vital signs, 
a task that can be facilitated using home-based non-intrusive IoT technology. In this 
context, our proposed system aims at extending the monitoring of patients’ physical 
activity levels using IoT technology, with the objective of detecting the impact of an 
inhibitor drug on the subject’s behavior changes over time. 


3 Methodology 


In this section, we first describe our methodology at the system implementation 
level where we are present our system architecture and our approach to deployment. 
Following, we present the data analysis algorithm we used in the data collection section. 


3.1 System Implementation 


Architecture Overview: This project relies on the AMI-Platform [28] solution we have 
developed. Data flows through the platform as follows: A cluster of heterogeneous 
sensors at the end-user layer collects environmental and biomedical data, such as tem- 
perature, heart rate, and location. This data is then sent to the physical gateway for 
pre-processing and transmitted to the cloud layer through a secure channel established 
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by the network layer. At the cloud layer, the data is decoded, processed, and stored in 
databases. The business layer serves as the user interface, offering dynamic graphs and 
charts for improved data visualization. 

Deployment: We setup two gateway nodes. The first node handles serial communi- 
cation with the sleep mat sensor that collects Ballistocardiogram (BCG), from which we 
can infer on the participant’s breathing rate and heartrate during sleep. The second node 
handles communication with all other deployed sensors that measures the participant’s 
indoor activity, including door sensors but also motion sensors that report on luminance, 
humidity, and temperature. 


3.2 Data Collection and Analysis 


In this study, a subject has been monitored for 7 months, from the end of July 2021 to 
early January 2022. The objective of this study is to detect changes in the overall behavior 
of the monitored subject after being given doses of a drug that inhibits physical activity. 
We have imputed missing activity values using the approach proposed by Ahmed et al. 
[29]. The overall activity curve for the subject inside his residence during the entire 
monitoring period is presented by Fig. 1. The timestamps at which these doses are given 
to the subject are tabulated in Table 1. 


Table 1. Timestamps at which actions were performed by the team. 


Timestamp Action performed 
27th July 2021 System installation 
3" August 2021 First visit 

26'" October 2021 Second visit 

6th January 2022 Last visit 


The detection algorithm measures the mean and standard deviation within a 3-days 
sliding window of activity calculated along the complete time series for the subject’s 
physical activity. Box-and-whisker plots were than calculated (Fig. 2) and used to define 
outliers for both the averages and standard deviations. Note that in our box-and-whisker 
plots outliers are calculated as all values above or below 1.5 times the interquartile 
range (the interquartile range where 50% of the data around the median). We defined 
outliers as change timestamps for the subject’s behavior. These outlier values are then 
back projected to the time domain to obtain the corresponding change timestamps. The 
corresponding change timestamps are presented in Table 2. The mean and standard 
deviation timestamps by both average and standard deviation at which the behavior 
changes are plotted with red dots overlapped on the same overall activity curve and 
presented by Fig. 2. 
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The change detected by the mean sliding window conforms with the expected times- 
tamps for the inhibition effect more than that detected by the standard deviation sliding 
window. This is because these timestamps are close enough in temporal domain to those 
timestamps at which the 3 doses are given to the subject. However, the standard deviation 
sliding window detects more change timestamps than the average sliding window does. 
Although the change is detected at the end of the window, the right change timestamp is 
corrected by the window size, i.e., three days are subtracted from the detected timestamp. 


4 Data Analysis Results 


Daily activity time series for the subject under the effect of an activity inhibiting drug is 
presented in Fig. 1. There are two behavior states that can be found along the time series, 
and both have a similar meaning. One is characterized by a relatively low standard devi- 
ation, we referred to this state as the background activity state. The other is characterized 
by high standard deviations, we refer to this state as an (low or high) activity induced 
state. The background activity state is dominant across the monitoring period while the 
activity induced state occurs for short period in mid-August 2021, at the beginning of 
November 2021 and during the first week of December 2021. The background activity 
state is characterized by activity values ranging approximately between 5000 to 7000 
movements (per day) with a standard deviation ranging between 0 and roughly 1800 
(Fig. 3. Subject behavior changes timestamps overlapped on the activity timeseries). 
While the activity induced state is characterized by activity values higher than 7000 or 
lower than 4000 movements (per day). 
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Fig. 1. Subject physical activity time series. 
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Fig. 2. Boxplot and outliers for both average and standard deviation sliding window. 


In our pilot study, 60% of the detected change (outliers) timestamps are associated 
with the standard deviation sliding windows’ outliers while the remaining 40% of the 
detected change timestamps are associated to the mean sliding. Note also that there is a 
change detection overlap of for 25% of the detected timestamps. There are six groups 
of change timestamps distributed along the entire daily physical activity timeseries of 
the subject (Fig. 3. Subject behavior changes timestamps overlapped on the activity 
timeseries). The 1% group starts in late July and ends in early August 2021, the 2"¢ 
group span the second half of August 2021, the 3™ group is in late October 2021, 
the 4™ group is at the beginning of November 2021, the 5" group is at the beginning 
of December 2021, and the 6" group is in late December 2021. The typical changes’ 
timestamps are tabulated in Table 2. 
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Fig. 3. Subject behavior changes timestamps overlapped on the activity timeseries. 
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Table 2. Detected changes’ timestamps and their corresponding values and source of detection. 


Detected Change Timestamp Outlier Value Source of Detection 

7/29/2021 6181 Average 

7/30/2021 10400 Average Standard Deviation 
7/31/2021 3750 Standard Deviation 

8/1/2021 4545 Standard Deviation 

8/19/2021 4227 Average 

10/30/2021 2337 Standard Deviation 

11/8/2021 3922 Standard Deviation 

11/9/2021 4877 Average 

12/4/2021 2834 Standard Deviation 

12/7/2021 508 Average Standard Deviation 
12/9/2021 3181 Average 

12/10/2021 7076 Average Standard Deviation 
12/11/2021 5463 Standard Deviation 

12/23/2021 7501 Average 

12/25/2021 7204 Average 


5 Discussion 


For the inhibition drug experiment the average activity level is very high; about 6000 
movements on average per day as the subject is a young person. Based on the timeseries, 
the subject’s activity level declines from late July 2021 to mid-August 2021. This decline 
is conformed with the expected outcome of the inhibition drug given to the subject as 
the effect of the drug is to inhibit the physical activity of the subject. This observation 
is in-line with the timestamp marked by the medical staff as the drug’s 1% dose intake, 
as reported in Table 1. There is another decline in the subject’s physical activity from 
October 30" 2021, to November 9" November 2021, which matches the administration 
of the 2"¢ dose as reported by the medical staff. A 3" decline in the subject’s physical 
activity is observed and detected during the period extending from late December 2021 
to early January 2022. In addition, a recovery period representing the subject’s convales- 
cence from the inhibition drug to his normal overall daily physical activity is observed 
from mid-August 2021 to early September 2021. A 2" recovery period is also observed 
in mid-November 2021. From the subject’s activity timeseries, we can observe that a 
convalescence period of roughly one week is required for the subject to recover from 
the effect of the inhibition drug. 
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6 Conclusion 


In this paper we presented a promising application of IoT to monitor inhibitor drugs. 
We proposed to monitor the behavior and activity of individuals under the influence of 
drugs to assess their effectiveness, identify side effects, and make timely adjustments 
to drug levels based on patient status. As such, we have presented a novel approach for 
monitoring and analyzing the behavior of a subject in an indoor environment over a period 
of seven months. Our approach relies on the loT AMI-Platform solution which collected 
data from a variety of sensors deployed in the participant’s environment. Our method uses 
sliding windows to calculate the mean and standard deviation of the subject’s activity over 
a 3-day period. Following, we rely on box-and-whisker plots to detect outliers in these 
values. These outliers represent changes in the subject’s behavior, which can be further 
analyzed to understand the effects of specific interventions, such as the administration 
of an inhibitor drug. 

Our results also show the effectiveness of our approach in detecting changes in 
behavior over a 7-month monitoring period. Specifically, we were able to observe the 
expected decline in the subject’s physical activity following the administration of the 
inhibitor drug, as well as recovery periods following each dose. Such information is 
useful to medical professionals to evaluate the effectiveness of the drug and making 
decisions about its administration in the future. They also highlight our platform as 
a valuable tool for deploying and analyzing data collected from sensor networks in a 
variety of applications, including in healthcare as presented in our case study. 
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Abstract. The role of medical test results in the diagnosis and treat- 
ment of a patient’s disease cannot be denied. Doctors and medical staff 
rely on these results to develop a treatment plan that meets the require- 
ments of the patient’s health status (i.e., physical condition) and disease 
type. In developing countries (i.e., Vietnam), we note that test results 
are recorded in paper versions and stored by patients. Thus, several solu- 
tions have been introduced for electronic medical records to compen- 
sate for medical test results management. However, in Vietnam, these 
approaches face many obstacles such as centralized processing (e.g., stor- 
age, analysis); non-transparency issues; scalability; availability; and so 
on. In this paper, we exploit the benefits of blockchain, smart contracts, 
and NFT technologies to solve the above disadvantages. Therefore, our 
work contributes to five aspects. (a) Collecting procedures for handling 
and storing test results of patients at hospitals in Ho Chi Minh City 
and Mekong Delta (i.e., Can Tho city) (b) Proposing a mechanism for 
sharing test results based on blockchain technology, smart contract, and 
NFT applied; (c) Presenting an NFT tool-based certification generation 
model; (d) implementing the proposed model based on smart contracts 
(i.e., proof-of-concept); and (e) deploying proof-of-concept on four EVM- 
and NFT-supported platforms to find the most suitable one. 


Keywords: Medical test result + Blockchain - Smart contract - NFT - 
Ethereum - Fantom - Polygon - Binance Smart Chain 


1 Introduction 


Building a decentralized patient information management system to be able 
to share them among the different medical centers is an urgent requirement 
today, not only in developed countries but also in other developing countries [8]. 
While developed countries have a specific set of laws and policies regarding the 
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sharing and use of personal/medical data of patients other than diagnostic and 
therapeutic purposes (eg, GDPR for European countries’, Privacy Act for Aus- 
tralia”), developing countries have not focused on synchronizing patient medical 
records [10]. In developing countries, personal data (i.e., including medical data) 
is not given due attention. Specifically, all information related to medical history 
and test results is stored locally at medical treatment facilities [6]. In addition, 
patients are not well aware of the process for handling and sharing their personal 
information - affecting patient privacy [5]. 

For medical data processing in developing countries (eg, Vietnam, India), most 
systems store information in the form of centralized storage. We also surveyed sev- 
eral hospitals in the Mekong Delta provinces (i.e., Ben Tre, Vinh Long) to observe 
how information about medical test results is stored. We found that the frontline 
hospitals in large cities store information (called electronic data) only focusing 
on the treatment route (i.e., patients undergoing internal medicine). inpatient) - 
for outpatient (i.e., home) tests or treatment results are recorded in paper form 
(called paper data - ie, the procedure is described in Sect. 3). 

Nevertheless, centralized storage carries with it many risks [18], for example 
easily being lost due to patient error or due to natural causes (eg, flood, fire). The 
loss of medical test results directly affects the diagnosis and treatment process. 
Hence, a solution is needed to store all this diagnostic information on electronic 
data. To solve this problem just focusing on infrastructure is not enough. Indeed, 
Sharma et al. [19] argue that centralized storage limits scalability and limits 
patient choice. We also note an example in Vietnam related to hospital transfer. 
Severe cases that cannot be treated at provincial hospitals (eg, Vinh Long, Ben 
Tre) are transferred to the larger hospitals in Ho Chi Minh City. Information 
related to testing results will be transferred based on paper documents (instead 
of directly retrieving from upper-level hospitals). 

To solve the above problems, several approaches have combined Blockchain 
technology and smart contracts to store and process medical data, respec- 
tively. For example, Xiao et al. [30] has proposed a system called HGD (Health 
Care Data Gateway). Thanks to the transparency and decentralized storage of 
blockchain technology, patients can easily and securely store and share their 
data. Similarly, Kassab et al. [11] has provided analysis to demonstrate how 
the benefits of blockchain technology (i.e., synchronous processing, decentral- 
ized storage) can enhance data processing and storage capacity for healthcare 
systems. In addition to the above two preliminary studies, we also summarize 
some research directions on the same topic based on blockchain technology and 
smart contract in the related work section of the research paper. 

One of the problems that the existing approaches have not solved (i.e., applied 
to developed countries) is the background of the users. In particular, it is not 
possible to require users (i.e., patients) to install a series of sharing policies for 
authorized users [22,28]. In addition, several studies have evaluated the feasibil- 
ity of storing all medical data on-chain [31]. Most of the above studies provide an 


1 https://gdpr-info.eu/. 
? https: //www.ag.gov.au/rights-and-protections/privacy. 


Towards a Medical Test Results Management System 81 


off-chain storage solution of information that is not directly related to the 
patient’s treatment. This has two purposes: (a) restricts on-chain storage - 
reduces the cost of query execution (i.e., data access) [4]; (b) restricts privacy 
violations because data stored on-chain is only accessible to users of the same 
system [16]. However, the above approach still faces problems when sharing their 
medical records with medical centers (i.e., medical staff: doctors, nurses, labora- 
tory staff). 

Our research problem in this paper is to fill the limitations on sharing medical 
data (i.e., medical test results) that apply to Vietnam. We have actually surveyed 
the testing process and its results storage of 2 large hospitals in Ho Chi Minh City 
(i.e., the University of Medicine and Pharmacy Hospital and Cho Ray Hospital) 
2 hospitals in Can Tho (i.e., central general hospital, the Children’s Hospital); 1 
hospital in Ben Tre (i.e., Nguyen Dinh Chieu Hospital); and 1 hospital in Vinh 
Long Province (i.e., Vinh Long General Hospital) in November 2022. Our records 
range from the time the patient gets tested to the time the results are received 
and treatment at the hospital. Therefore, our work contributes on five aspects, 
namely (a) collecting procedures for handling and storing test results of patients 
at hospitals in Ho Chi Minh City and Mekong Delta (b) proposing a mechanism 
for sharing test results based on blockchain and smart contract technologies; (c) 
presenting an NFT tool-based generation model for storing medical test results; 
(d) providing the proof-of-concept of the proposed model; and (e) deploying 
our implement on 4 (ERC721 and ERC20)-supported platforms including BNB 
Smart Chain, Fantom, Polygon, and Celo to choose the most suitable platform 
for our proposed model.® 


2 Related Work 


To the best of our knowledge, most of the articles focus on proposing a model 
to protect personal information and medical data based on two approaches (i) 
authorization (i.e., access control) and blockchain technology. In this section we 
categorize state-of-the-arts based on the above two themes. Thereby, analyze the 
shortcomings in the current work and propose our model. 


2.1 The Approaches Based on Access Control Models 


Most state-of-the-arts exploit access control models (RBAC, ABAC, ABE) to 
ensure access to patient medical data. In particular, data retrieval requests must 
be approved by the data owner (i.e., patient). For example, Makubalo et al. [14] 
has synthesized access control model-based approaches to empowering patients 
to share medical information. Specifically, policies will help patients manage 
who (eg, nurses, doctors) is allowed to access their medical data and use it for 
a specific purpose (eg, treatment). Another example applying attribute-based 


3 We do not deploy smart contracts on ETH because the execution fee of smart con- 
tracts is too high. 
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encryption (ABE) was introduced by Barua et al. [1] and Yin et al. [32] to 
encrypt all medical data before storing it in the cloud. Only objects that satisfy 
the required attribute have the key to decrypt them. In addition, based on the 
Role-based Access Control model Chen et al. [2] described a requester role-based 
access management model. Besides, some methods apply dynamic policy model 
to increase flexibility in medical environment [9,21] or design two layers of policy, 
ie, i) security policy - protect data patient data from intrusion by users outside 
the system, and ii) privacy policy that protects patient information from agents 
in the system [28]. Combining dynamic policy-based approaches and the ABAC 
model, son et al. introduced a two-layer access control model based on medical 
environments [22]. 


2.2 The Approaches Based on Blockchain Technology 


Chen et al. [3] proposed a model based on Blockchain technology applied to health- 
care environment to store and control mining data from IoT devices. These devices 
store real-time medical data. All information stored on the chain is authenticated 
and controlled by the user. Madin et al. [13] has proposed a model of storing med- 
ical record information on blockchain and storing details of that information on 
IPFS. The purpose of this group of authors is to protect the privacy of patient 
data (i.e., limit access from users of the same system). MedAccess introduced the 
patient Health Record management system [15]. The proposed system focuses on 
protecting patient information from being altered (i.e., transparency) and privacy 
(offchain storage). Similar to the approach of [15], to minimize the amount of infor- 
mation stored on the chain, Zyskind et al. [33] presents an onchain and offchain 
patient data and information storage model. 

For actual deployed solutions, HealthBank* combined blockchain and IPFS 
to propose a patient-centric model for sharing medical data. Several solutions are 
introduced to comply with privacy constraints (eg, GDPR). For example, Health- 
Nautica and Factom Announce Partnership” presented a decentralized data stor- 
age solution. and protect personal information based on Blockchain technology. 
Another approach based on data streaming is also proposed by IRYO® based on 
NuCypher KMS (ie [7]). 

All of the above evaluation directions face two major challenges for both the 
access control model-based approach and the Blockchain and IPFS-based app- 
roach. Specifically, with the first approach of empowering users based on access 
policies, the approaches are limited when users have to create a series of policies to 
control access accordingly. These policies are prone to conflicts or redundancies if 
the number of access objects is large and there are many properties [28]. Moreover, 
getting used to a completely new system (i.e., defining security policies) also makes 


* https: //www.healthbank.coop/2018/10/30/healthbank-creates-the-first-patient-cen 
tric-healthcare-trust-ecosystem/. 

5 https: //www.factom.com/company-updates/healthnautica-factom-announce-partn 
ership/. 

6 https: //iryo.network /iryo_whitepaper.pdf. 
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users encounter some obstacles due to lack of necessary background (especially for 
casual users) in developing countries. As for the second approach (blockchain and 
IPFS-based), it is agreed that this method brings a lot of efficiency in the pro- 
cess of storing and processing medical data, but on-chain data storage solutions 
and out-of-string also brings a lot of complications for the average user when they 
want to share. To limit the above risks, within the framework of the problem of 
sharing medical test results at medical centers in Vietnam, we combine a series of 
modern platforms including blockchain, smart contracts, and NFT. Specifically, 
instead of sharing policies to define groups of people who can retrieve test results or 
patient history, we create corresponding NF Ts. This approach restricts the strict 
requirements of the technology platform such as security policy-based methods. 
The next section presents our proposed model based on NFT technology to share 
information with the respective objects. 


3 The Blockchain-Based Medical Test Results 
Management System 


In this section, we summarize the traditional test results management process 
applied in Vietnam (i.e., in provinces and cities in the Mekong Delta) as well as 
propose a model based on blockchain technology. and smart contracts to solve 
the limitations of the traditional model as well as NFT technology to generate 
electrical test results for easy sharing in the medical environment. 


3.1 Traditional Medical Test Results Management Model 


if 


Medical staffs 


Nurses Mey. (2) 
og 
Vh 


Patients Medical record Laboratory 
book 


Fig. 1. Traditional medical test results management model 


Figure 1 shows the traditional process of testing and getting results based on 
four steps (i.e., depending on the hospital in the city or the country, the process 
is different). At provincial/major city hospitals, the waiting time to get tested 
and receive test results is very long. In particular, according to our perception 
at the University of Medicine and Pharmacy Hospital in Ho Chi Minh City. In 
Ho Chi Minh City, some patients have to be present from the hospital from 1:00 


84 N. T. Anh et al. 


AM to queue to receive the order number even though the opening time there 
is 8:00 AM. Therefore, the procedure shown in Fig. 1 shows only the general 
steps and omits the detailed steps depending on the hospital. Specifically, the 
patient creates an archive of health information called medical record book at the 
medical center.’ The patients undergo the necessary medical examinations before 
proceeding with the treatment. This information is extremely important - the 
doctor’s diagnoses and judgments are based on the changes in the patient’s health 
status through the test results. After receiving the results from the laboratory 
staff, the patient brings the results to the doctors and nurses to make judgments 
about their health status and corresponding treatments. All information about 
the diagnosis and treatment results are updated in the medical record book. 
Therefore, the loss of the patient’s medical record book seriously affects the 
treatment process. Therefore, there is a need for a comprehensive solution to the 
problems related to the storage and sharing of patients’ medical record books for 
both electronic and paper formats. The following section presents our solution 
based on current popular technologies: blockchain, smart contract and NFT 
applied to the Vietnamese environment. 


3.2 Medical Test Results Management Model Based on Blockchain 
Technology, Smart Contract and NFT 
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Fig. 2. Medical test results management model based on blockchain technology, smart 
contract and NFT 


Figure 2 shows nine steps to build a medical test result management system based 
on blockchain technology, smart contract and NFT. Specifically, the patient must 


T According to our records, most patients are given a paper version of the medical 
record book - one A number of medical conditions and treatment history or medical 
records are collected and stored on a local server. 
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create a global patience ID that is shared by the entire medical system in Viet- 
nam (step 1). According to our observations, the role of patience_ID global is not 
considered in the current system at the medical facility in Vietnam. Specifically, 
each patient can only receive one identification code (i.e., valid for the day) at 
each different hospital. The storage of patient information in medical facilities is 
not supported. Step 2 Synchronize patience ID global with the medical record 
book to store all information about the medical record and test results as well as 
the patient’s medical history. This information is displayed on the user interface 
based on our provided services (UI services) in step 3. Other users can access 
patient information to get a list of test requests. of the patient (step 4); update 
test results (step 5); or get feedback on your doctor/nurse’s medical condition, 
diagnosis, and treatment steps (step 6). All these steps are performed and man- 
aged based on pre-designed methods in smart contracts (step 7) and updated 
to the distributed ledger (step 8). The content of the NFTs is created in step 9 
(i.e., the content of the NFTs - medical test results are presented in the Intro- 
duction) and shared with doctors and nurses for the purpose of patient care and 
treatment. 


4 Evaluation Scenarios 


4.1 EVM-Supported Platforms 


Transactions Internal Txns BEP-20 Token Txns ERC-721 Token Txns Contract Events 


JF Latest 3 from a total of 3 transactions 


Txn Hash Method@ Block Age From T To Y Value [Txn Fee] 
© 0x0d2615893127314da4. Transter 24867386 1 day 17 hrs ago Oxcaa9c5b45206e083f4f. iN Œ Oxafa3888d1dfbfe957b1... OBNB  0.00057003 
® 0xifb5ae508ae1c00322. Mint 24867381 1 day 18 hrs ago Oxcaa9c5b45206e083f4f. N Œ Oxafa3888d1dfbfe957b1... OBNB  0.0010916: 
® —0x8b033161 df67984cc6, oxeososo40 24867375 1 day 18 hrs ago Oxcaade5b45206e083/4f m Contract Creation OBNB 0,0273134 


CSV Export $ 


Fig. 3. The transaction info (e.g., BNB Smart Chain) 


Because the model for generating medical test results makes it easy for patients 
to manage test results (i.e., can be extended to medical records) as well as eas- 
ily share them with other objects that users want. We want instead of defining 
security policies (eg, access control), we implement the proposed model on EVM- 
enabled blockchain platforms instead of exploiting Hyperledger eco-system plat- 
forms because they are easy to open. wide (i.e., using existing platforms and sys- 
tems). In addition, assessments based on system responsiveness (i.e., number of 
requests responded successfully /failed, system latency - min, max, average) have 
been evaluated by us in the previous research paper. Therefore, in this paper we 
determine the suitable platform for our proposed model. Specifically, we install 
a recommendation system on four popular blockchain platforms today, support- 
ing Ethereum Virtual Machine (EVM), including Binance Smart Chain (BNB 
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Smart Chain); Polygon®; Fantom!?; and Celo!!. Our implementations on these 
four platforms are also shared as a contribution to the article to collect trans- 
action fees corresponding to the four platforms’ supporting coins!?, ie, BNB!°; 
MATIC"*; FTM?; and CELO!®. For example, Fig. 3 details our three assess- 
ments of a successful installation on BNB Smart Chain (i.e., similar settings are 
shown for the other three platforms). Our implementations to evaluate the exe- 
cution cost of smart contracts (i.e., designed based on Solidity language) run on 
testnet environments of four platforms in order to choose the most cost-effective 
platform to deploy. Our detailed assessments focus on the cost of performing 
contract creation, NFT generation, and NFT retrieval/transfer presented in the 
respective subsections related to i) Transaction Fee; ii) Gas limit; and iii) Gas 
used by transaction. 


4.2 Transaction Fee 


Table 1. Transaction fee 


Contract Creation Create NFT Transfer NFT 

BNB Smart Chain | 0.0273134 BNB ($8.43) 0.00109162 BNB ($0.34) 0.00057003 BNB ($0.18) 

Fantom 0.00957754 FTM 0.000405167 FTM 0.0002380105 FTM 
($0.001849) ($0.000078) ($0.000046) 

Polygon 0.0068407 10032835408 0.000289405001852192 0.000170007501088048 
MATIC($0.01) MATIC($0.00) MATIC($0.00) 

Celo 0.007097844 CELO ($0.004) | 0.0002840812 CELO ($0.000) 0.0001554878 CELO ($0.000) 


Table 1 shows the cost of creating contracts for the four platforms. It is easy 
to see that the highest transaction fee of the three requirements is contract 
creation for all four platforms. In which, the cost of BNB Smart Chain is the 
highest with the highest cost when creating a contract is 0.0273134 BNB ($8.43); 
whereas, the lowest cost recorded by the Fantom platform with the highest cost 
for contract initiation is less than 0.00957754 FTM ($0.001849). Meanwhile, the 
cost to enforce Celo’s contract initiation requirement is lower than Polygon’s 
with only $0.004 compared to $0.01. For the remaining two requirements (Create 


8 https: //github.com/bnb-chain/whitepaper /blob/master/WHITEPAPER.md. 

? https: //polygon.technology /lightpaper-polygon.pdf. 

° https: //whitepaper.io/document /438/fantom-whitepaper. 

1 https://celo.org/papers/whitepaper. 

? Implementation of theme models our release at 11/24/2022, 8:44:53 AM UTC. 

3 https: //testnet.bscscan.com /address/Oxafa3888d1dfbfe957b1cd68c36ede4991e104a 
53. 

4 https://mumbai.polygonscan.com/address /Oxd9ee80d850ef3c4978dd0b099a45a559f 
d7cdef4. 

5 https://testnet.ftmscan.com/address/0x4a2573478c67a894e32d806c8dd23ce8e26f78 
47. 

6 https://explorer.celo.org/alfajores/address/0x4a2573478C67a894E32D806c8Dd23E 
E8E26f7847 /transactions. 
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NFT and Transfer NFT), we note that the cost of implementing them for all 
three platforms, Polygon, Celo, and Fantom is very low (i.e., negligible) given the 
cost. trades close to $0.00. However, this cost is still very high when deployed on 
BNB Smart Chain with 0.00109162 BNB ($0.34) and 0.00057003 BNB ($0.18) 
for Create NFT and Transfer NFT, respectively. 


4.3 Gas Limit 


Table 2. Gas limit 


Contract Creation | Create NFT | Transfer NFT 
BNB Smart Chain | 2,731,340 109,162 72,003 
Fantom 2,736,440 115,762 72,803 
Polygon 2,736,284 115,762 72,803 
Celo 3,548,922 142,040 85,673 


Table 2 shows the gas limit for each transaction. Our observations show that the 
gas limits of the three platforms (i.e., BNB, Polygon, and Fantom) are roughly 
equivalent - where Polygon and Fantom are similar in all three transactions. The 
remaining platform (i.e., Celo) has the highest gas limit with 3,548,922; 142,040; 
and 85,673 for all three transaction types. 


4.4 Gas Used by Transaction 


Table 3. Gas Used by Transaction 


Contract Creation | Create NFT Transfer NFT 
BNB Smart Chain 2,731,340 (100%) | 109,162 (100%) | 57,003 (79.17%) 
Fantom 2,736,440 (100%) | 115,762 (100%) 68,003 (93.41%) 
Polygon 2,736,284 (100%) | 115,762 (100%) 68,003 (93.41%) 
Celo 2,729,940 (76.92%) | 109,262 (76.92%) 59,803 (69.8%) 


Table 3 shows the amount of gas used when executing the transaction (i.e., what 
percentage of gas in total gas is shown in Table 2). Specifically, three platforms 
BNB, Polygon, and Fantom use 100% of Gas Limit for two transactions Contract 
Creation and Create NFT. Meanwhile, Celo uses 76.92% of the Gas limit for the 
above two transactions. For the last transaction of Transfer NFT, the highest 
Gas level was recorded by Fantom and Polygon with 93.41% of Gas limit; while 
BNB and Celo use 79.17% and 69.8% of Gas limit. 
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5 Discussion 


According to our observation, the transaction value depends on the market 
capitalization of the respective coin. The total market capitalization of the 
4 platforms used in our review (i.e., BNB (Binance Smart Chain); MATIC 
(Polygon); FTM (Fantom); and CELO (Celo)) are $50,959,673,206 respectively. 
$7,652,386, 190; $486,510,485; and $244,775,762.!" This directly affects the coin 
value of that platform - although the number of coins issued at the time 
of system implementation also plays a huge role. The total issuance of the 
four coins BNB, MATIC, FTM, and CELO is 163,276,974/163,276,974 coins; 
8,868,740,690/10,000,000,000 coins; 2,541,152,731/3,175,000,000 coins and 473, 
376,178/1,000,000,000 coins. The coin’s value is conventionally based on the 
number of coins issued and the total market capitalization with a value of 
$314.98; $0.863099; $0.1909; and $0.528049 for BNB, MATIC, FTM, and CELO 
respectively. 

Based on the measurements and analysis in 4 section, we have concluded that 
the proposed model deployed on Faltom brings many benefits related to system 
operating costs. In particular, generating and receiving NFTs has an almost 
zero (i.e., negligible) fee. Also, the cost of creating contracts with transaction 
execution value is very low (i.e., less than $0.002). In future work, we plan to 
implement complex methods/algorithms (i.e., encryption and decryption) and 
more complex data structures to observe the costs for the respective transactions. 
Deploying the proposed model in a real environment is also a possible approach 
(i.e., implementing the recommendation system on the FTM mainnet). In our 
current analysis, we have not considered issues related to the privacy policy of 
users [23] (i.e., access control [21,22], dynamic policy [20,29]) - a possible app- 
roach would be implemented in upcoming work. Finally, infrastructure-based 
approaches (i.e., gRPC [12,25]; Microservices [17,26]; Dynamic transmission 
messages [27] and Brokerless [24]) can be integrated into the model of us to 
increase user interaction (i.e., API-call-based approach). 


6 Conclusion 


Our article aims at a solution to share test result data in the medical environment 
based on the benefits of current technologies such as blockchain, smart contracts, 
and NFT. Our solution aims to replace the paper documents used in hospitals in 
the South of Vietnam, including hospitals in Ho Chi Minh City. Ho Chi Minh and 
the Mekong Delta. Specifically, these results are stored off-chain and generate 
corresponding NFTs upon request from treatment facilities. To reduce patient 
effort, we replace policy-based approaches with data-sharing solutions for disease 
care and treatment. Our solution is implemented on the Ethereum platform and 
Solidity language (i.e., smart contract development). In order to optimize the 
execution cost of basic transactions (i.e., contract creation, NFT creation, NFT 
transfer), we deploy proof-of-concept on four EVM-enabled platforms, including 


17 Our observation time is 12:00PM - 11/26/2022. 
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BNB, MATIC, FTM, and CELO. According to our observations (i.e., executed at 
the time of system evaluation), smart contracts deployed on Fantom bring more 
economic value (i.e., cost savings in execution) than the other three platforms. 
The future directions of the current work are presented in the Discussion section. 
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Abstract. Children diagnosed with Autism Spectrum Disorder (ASD) 
often exhibit agitated behaviors that can isolate them from their peers. 
This study aims to examine if wearable data, collected during every- 
day activities, could effectively detect such behaviors. First, we used the 
Empatica E4 device to collect real data including Blood Volume Pulse 
(BVP), Electrodermal Activity (EDA), and Acceleration (ACC), from a 9- 
years-old male child with autism over 6 months. Second, we analyzed and 
extracted numerous features from each signal, and employed different clas- 
sifiers including Support Vector Machine (SVM), Random Forest (FR), 
eXtreme Gradient Boosting (XGBoost), and TabNet. Our preliminary 
findings showed good performance in comparison with the state of the 
art. Notably, XGBoost demonstrated the highest performance in terms 
of accuracy, precision, recall, and F1-score. The accuracy achieved in this 
paper using XGBoost is equal to 80% which exceeds previous research. 


Keywords: ASD - Agitated behaviors - Wearable data - Features 
extraction - Classification 


1 Introduction 


Autism Spectrum Disorder (ASD) is a neurodevelopmental disability charac- 
terized by persistent difficulties in social communication and interaction [1]. 
Patients with ASD could display repetitive patterns of challenging behaviors 
such as self-injury, meltdown and tantrum, and aggression [2] which arise from 
a set of triggers. 

The agitation has negative outcomes for both ASD patients and their care- 
givers. It can lead to increased stress levels and impaired quality of life [3]. 
Consequently, the need for assistive technologies to improve the well-being of 
children with ASD and their caregivers is growing. With the advance of technol- 
ogy, we could develop solutions to prevent these maladaptive behaviors. Before, 
behaviors were reported by caregivers. In such cases, the physical presence of 
© The Author(s) 2023 
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the caregiver with the child during his daily activity is required. In this regard, 
the development of automatic detection methods based on wearable signals is 
needed to replace this constant physical assistance. Recently, the detection of 
agitated behaviors based on wearable data has grabbed the attention of scien- 
tists. Although, there is a lack of literature related to this topic. 

Given the above-mentioned, this paper explores the feasibility of detecting 
agitated behaviors of ASD children based on their physiological and kinetic sig- 
nals. We will compare different classification algorithms (SVM, Random Forest, 
XGBoost, and TabNet). The main contributions of our study are: 


— Collection of real data from a 9-year-old male autistic child during his every- 
day activities. Compared to previous works, our data collection is performed 
in a natural and real-life process. 

— Investigation of numerous features from electrodermal activity, blood volume 
pulse, and acceleration signals. 

— Development of various classifiers resulting in improved performance com- 
pared to the state-of-the-art methods. 


The remainder of this paper is as follows: Sect. 2, provides a literature review. 
We will give more details about our proposed models 3. Section 4 discusses our 
results and compares them to previous research. Finally, Sect.5 concludes this 


paper. 


2 Literature Review 


In this section, We will first define EDA and BVP signals, then we will review 
related work in the subsequent section. 


2.1 Physiological Signals 


Biomedical signals are very rich with information about changes in the psy- 
chophysiological systems. Hence, we will investigate the performance of some 
physiological signals to detect agitated behaviors of ASD children, basically EDA 
and BVP signals. 


Electrodermal Activity: also known as Galvanic Skin Response GSR, reflects 
the autonomic changes in the electrical and conductivity properties of the skin 
consequent to an increased sweat glands activity [4,5]. EDA has been widely used 
as a physiological index in psychophysiological research since it is linked to auto- 
nomic emotional and cognitive processes. EDA serves as a valuable biomedical 
indicator in many fields such as emotions and behaviors [6]. 

EDA is an aggregate of two components: the tonic activity (Skin Conduc- 
tance Level SCL) and the phasic activity (Skin Conductance Response SCR) [4]. 
Several methods have been developed to extract SCR and SCL from EDA. In 
this paper, we will use the cuzEDA [7] which relays on convex optimization and 
maximum a posteriori approach. 
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The phasic component corresponds to a series of SCR peaks typically char- 
acterized by three components: onsets, peaks amplitude, and half-recovery. Some 
methods have been suggested to identify these peaks and their characteristics. 
In our case, we used the method described in [8] to detect these peaks from SCR. 


Blood Volume Pulse: is measured using the PPG sensor, which relays on 
the pulse oximeter technique to measure the blood volume in the micro-vascular 
of the target tissue [9]. In fact, the sensor illuminates the surface of the skin 
with IR light and measures changes in the back-scattered light and the portion 
absorbed by the skin [10]. Even though this technique is inexpensive and suitable 
for daily life applications, the BVP signal is susceptible to several artifacts. 
Nevertheless, health-related features such as heart rate and heart rate variability 
can be extracted from BVP. 

Basically, the signal has three characteristic points: diastolic point, systolic 
point, and dicrotic notch. 

In our study, we will use the NeuroKit [26] and HRV analysis [25] Python 
packages to process EDA and BVP signals. These packages offer comprehensive 
tools for the analysis and interpretation of physiological data. 


2.2 Related Works 


The literature on the detection of agitated behaviors using biomedical data is 
limited. While there have been studies investigating similar areas such as emotion 
recognition [11,12], anxiety, and stress detection [13] of ASD patients. 

Alban et al. [14] developed three machine learning models (Decision Trees, 
SVM, Multi-Layer Perceptron) to detect challenging behaviors of a 10-year-old 
male ASD child in interaction with social robots and toys based on physiological 
and kinetic signals. From each signal, they extracted four time-domain features 
(mean, min, max, and std) from a window of two seconds. To balance the dataset, 
they applied resampling. Combining all the signals and using the MLP, they 
achieved the best accuracy equal to 97%. Although the result reflects a good 
performance, it suffers from subjectivity to the experimental protocol of the 
study (interaction with social robots and toys). 

Meltdowns and tantrums often lead to impulsivity and agitation. Khullar et 
al. [15] designed a wristband with three sensors to collect Heart Rate (HR), Skin 
Temperature (ST), and Galvanic Skin Response (GSR) in order to detect melt- 
downs and tantrums in ASD patients. Different preprocessing techniques were 
applied (null values removal, filtering, normalization). A CNN+LSTM model 
was developed to detect these behaviors. The performance in terms of precision, 
recall, and F-1 is equal to 0.98, 0.95, and 0.97 respectively. Unfortunately, the 
authors did not provide any detail about the ASD patients or the data collection 
protocol. 

Other studies have developed models to predict atypical behaviors. For 
instance, Goodwin et al. [16] developed a predictive model to anticipate agi- 
tated behaviors 1-min before they occur using physiological and kinetic signals. 
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Their study involved 20, ASD patients. The signals were filtered and processed to 
extract 10 time-domain statistical features, and 2 binary aggression labels were 
also extracted and used to train a ridge-regularized logistic regression model. 
Two types of models were investigated: a global model using data from all par- 
ticipants, and 20-person dependent models, developed for each patient. The Area 
Under the Curve AUC of the global model is 0.71, while the average AUC for 
person-independent models was 0.84. 

In a second study, Goodwin et al. [17] improved their previous models by 
using Principal Component Analysis PCA for features reduction and kernel- 
based SVM to predict aggressive behaviors 3 min in advance. The global model 
achieved an increased AUC of 0.98. 

The major drawback of the aforementioned studies is that the datasets were 
collected in controlled environments under a specific experimental setup. Con- 
sequently, the results are subjective to the experimental protocols. In contrast, 
our study focuses on developing detection models for agitated behaviors in a 
real-world context during the everyday activities of an ASD child. 


3 Proposed Models 


3.1 Data Collection 


It is important to note that there is currently no publicly available dataset for 
this specific topic. Consequently, the first step consists in collecting real data 
from an ASD child using the Empatica E4. 

We were able to recruit a 9-year-old male autistic child to take part in our 
experiments, although many families we contacted refused to participate. The 
patient suffers from moderate to severe autism and does not take any medication. 
Data collection took place over a period of 6 months during his daily activities 
(playing outside, doing homework, etc.). 

E4 records EDA (4 Hz), BVP (64 Hz), Heart Rate HR (1 Hz), Inter Beat 
Intervals IBI, Skin Temperature ST (4 Hz), and acceleration ACC along the z, 
y, and z—axis (32 Hz). The wristband has an internal 60 hours memory capacity. 
Data transfer between E4 and the computer is made via an E4 manager Software 
that should be installed on the computer. The recorded signals are exported to 
csv files for further analysis. 

The child wore the device and performed his activities in a naturalistic man- 
ner. The parents accompanied their child to annotate the moments of agita- 
tion. It should be noted that the participant refused to wear the device during 
school, resulting in most of the activities being conducted at home. Therefore, 
the moments of agitation were relatively rare compared to normal behaviors. In 
fact, the child acts aggressively at school or when confronting other people. 


3.2 Signal Processing 


The quality of signals significantly impacts the performance of a model. Mainly, 
in this step, we will apply filters to reduce the noise from raw signals, followed by 
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the extraction of relevant features. The collected signals are corrupted by differ- 
ent sources of artifacts. The following paragraphs detail the distinct processing 
steps employed to extract relevant features from raw signals. 


EDA Processing: EDA is hampered by its sensitivity to motion artifacts [19]. 
Raw EDA is firstly pre-processed using the wavelet-based-adaptive denoising 
procedure as described in [19]. Secondly, the signal is filtered using 4t” order But- 
terworth low-pass forward-backward filter with Fout = 0.5 Hz as in [20]. Thirdly, 
we employed min-max normalization to remove the difference between the differ- 
ent recordings. Moving forward, the next steps include the decomposition of EDA 
into SCR and SCL using the cvxEDA algorithm [7]. The three components are 
segmented into overlapping windows for features extraction [20,21]. In contrast 
to papers reviewed previously that have investigated a limited set of statistical 
features from raw EDA, our work encompasses a wider range of features from 
raw EDA, SCR, and SCL. More specifically, we extract time-domain features, 
peak-related features, and frequency-domain features. In total, we calculate 80 
distinct features, which are summarized in Table 1. 


Table 1. Extracted features from EDA, SCR, and SCL. 


Category Name 


Time domain 1. mean, 2. std, 3. min, 4. max, 5. range of raw EDA, SCR, and SCL 

6. slope of raw EDA, 7. energy, 8. sum, 9. skew, 10. kurtosis, 11. IQR, 
12. integral, 13. iqr5-95, 14. 5‘”, 15. 95'” percentile, 16. # of points 
below/17. above the mean, 18. permutation/ 19. svd entropy of 

SCR, and SCL, 20. Pearson correlation between time and SCL, 

21. mean (scr), 22. std(Ascr), 23. mean(OZcR), 24. std(dcr), 

25. arc length, 26. norm average power, 27. norm RMS of SCR 

Peaks features 28. # of SCR peaks, 29. mean, 30. std, 31. min, 32. max of peaks, 

33. mean, 34. std, 35. min, 36. max of onsets, the sum of 37. SCR 
duration response, 38. SCR amplitude, 39. the area under SCR response 


frequency domain | 40. 5 spectral power magnitude, 41. total power magnitude in 
[0-0.5 HZ] band of raw EDA, SCR, and SCL 


BVP Processing: The BVP signals of our recordings are very noisy. We first, 
used the winsorization technique to remove values above 98°” and below 254 
percentile as in [20]. Next, we applied a 6” order Butterworth band-pass forward- 
backward filter with a frequency range [0.6Hz — 3.3Hz] as in [22]. Later, we 
normalized the cleaned BVP using min-max. To extract systolic points, we used 
the pipeline described in [23], which are later cleaned using the HRV-analysis 
[25] to discard outliers and ectopic peaks [24]. RR intervals are calculated from 
BVP based on systolic points. The different types of features extracted from the 
RR interval basically comprise time-domain, frequency-domain, geometrical, and 
Poincare-plot features. The following Table 2 defines the 30 BVP features. 
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Table 2. Extracted features from BVP signal. 


Category Name 

Time domain 1. mean, 2. std of HR, 3. mean, 4. std, 5. skew, 6. kurtosis 

of HRV, 7. # and 8. % of HRV intervals differing more than 
50 ms and 20 ms, 9. mean, 10. median, 11. std, 12. RMSSD, 
13. kurtosis, and 14. the skew of relative RR intervals, 

15. RMSSD, 16. SDSD, 17. SDSD/RMSSD of all intervals, 

18. difference between adjacent RR intervals, 

Geometrical 19. triangle index of HRV, 

Poincare-plot 20. SD1, and 21. SD2 term Poincare-plot descriptor of HRV, 
Frequency domain | 22. VLF(0.003-0.04 Hz), 23. LF(0.04—-0.15 Hz), 24. HF(0.15-0.4 Hz) 
band in the HRV power spectrum and their normalized values, 
25. LF/HF of HRV, 26. total power in [0.003 Hz—0.4 Hz] 


ACC Processing: We filtered the ACC signals as suggested by [27]. We applied 
a low-pass 4” order Butterworth filter with F..4 = 10 HZ. Features extracted 
from each axis include time-domain and frequency-domain features as summa- 
rized in Table 3. The total number of features extracted from ACC data is 69. 


Table 3. Extracted features from ACC signals. 


Category Name 

Time domain 1. mean, 2. std, 3. range, 4. sum, 5. skew, 6. kurtosis, 7# of peaks, 
8. RMS, 9. integral, 10 # above the mean, 11. # below mean, 

12. # of change sign, 13. IQR, 14. iqr5-95, 15. 5", 16. 95°" pct 
17. permutation entropy, 18. svd entropy 

Frequency domain | 19. 3 spectral power magnitudes in the [1-5.5] Hz bands 

20. total power magnitude in the [1-5.5] Hz band 


3.3 Methodology 


Our main goal is to detect agitated behaviors of ASD patients using wearable 
data. For this purpose, we will establish different binary classifiers. While, as 
stated above, the dataset collected suffers from severe imbalance. For skewed 
class distribution, we typically apply cost-sensitive learning models [28]. This 
is because misclassifying agitated behaviors carries a higher cost compared to 
misclassifying normal behaviors during the model training process. 

The first step after collecting the data consists in processing the different sig- 
nals and extracting salient features. Subsequently, we will explore several types 
of classifiers: Support Vector Machine [30], Random Forest [29], XGBoost, and 
TabNet [32]. SVM works by finding a hyperplane that separates the classes of the 
training data. Random Forest is an ensemble learning algorithm based on mul- 
tiple decision trees each of which is trained on a different subset of the training 
data and a random subset of features. XGBoost follows the principle of boosting 
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[31] which consists in combining a set of weak classifiers to create a stronger one. 
TabNet on the other hand, is a deep-learning model designed for tabular data. 
It uses a sequence of attention mechanisms to evaluate the importance of each 
feature at every training step. 


4 Experiments 


4.1 Setup 


Our dataset comprises multiple sessions, with the child exhibiting aggressive 
behaviors in only 6 of the recordings, and these behaviors were observed for a 
few minutes. To extract features, we segmented the recordings into windows of 
30 seconds with a 50% overlap. For each window, we calculated the features illus- 
trated in the previous Sect.3.2. The next step consists in training the different 
models over a dataset containing segments from all sessions. To ensure optimal 
model performance, we conducted an extensive search for the best hyperparam- 
eters for each model using the Grid Search method. The hyperparameters that 
yielded the best results are presented below: 


e SVM: kernel =rbf, gamma = scale, and C = 5. 

e RF: n_estimators = 1000, criterion = gini, max_depth = 20, max features = 
auto, min-samples-_split = 5 and min_samples_leaf = 4. 

e XGBoost: n_estimators = 2500, eta = 0.01, max_delta_step = 8, max_depth 
= 12, min-child_weight = 12, gamma = 0.5, reg_alpha = 0.1 

e TabNet: n-d, n-a = 64, n_steps = 6, gamma = 1.5, lambda_sparse = 0.01, 
learning_rate = 0.025, momentum = 0.98, max_epochs = 300. 


4.2 Evaluation Metrics 


We used stratified 10—fold cross-validations to test the performance of our mod- 
els since it keeps the same dataset imbalance proportion. We calculated precision, 
recall, F-1 score, and balanced accuracy. Furthermore, we calculated the AUC 
and Precision-Recall AUC (PR AUC). AUC measures the model’s performance 
by calculating the True Positive rate against the False Positive rate. On the 
other hand, PR AUC measures the precision and recall of a model. This metric 
is useful when the model suffers from imbalance. 


4.3 Results 


In this section, we will present the results achieved by each classifier and compare 
them to previous research. 
Table 4 corresponds to the average evaluation metrics values using the differ- 
ent classifiers and combining the features from EDA, BVP, and ACC signals. 
These results revealed that the XGBoost model outperformed RF, TabNet, 
and SVM, achieving an 80% balanced accuracy. Specifically, precision, recall, 
F1-score, AUC, and PR AUC of XGBoost are 0.77, 0.60, 0.67,0.80, and 0.72, 
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Table 4. Average evaluation metrics of the stratified 10-fold of the different classifiers 


using the combined features (Mean% 4 


E Standard Deviation%) 


Metrics XGBoost RF TabNet SVM 
Precision 0.77 + 0.08 | 0.34 + 0.05 | 0.59 + 0.11 0.77 + 0.11 
Recall 0.60 + 0.07 | 0.57 + 0.04 | 0.54 + 0.08 | 0.38 + 0.10 
F1-score 0.67 + 0.06 | 0.41 + 0.04 | 0.56 + 0.08 | 0.50 + 0.10 
AUC 0.80 + 0.03 | 0.76 + 0.02 | 0.76 + 0.04 | 0.70 + 0.04 
PR AUC 0.72 + 0.07 | 0.43 + 0.06 | 0.53 + 0.07 | 0.61 + 0.07 
Balanced Accuracy 0.80 + 0.03 | 0.76 + 0.02 | 0.77 + 0.04 | 0.70 + 0.05 


respectively. SVM achieved similar precision as XGBoost (0.7) while the recall 
is notably lower (0.38) resulting in a lower F1-score of 0.50. In terms of PR AUC, 
SVM achieved 0.61. RF model achieved the lowest average F1-score (0.41) and 
PR AUC (0.43). In the literature, TabNet outperformed XGBoost in several 
competitions including Kaggle, in our study, XGBoost yielded the best perfor- 
mance. One possible reason could be the limited amount of data to train TabNet. 
Nevertheless, it still achieved the second highest F1-score, AUC, and balanced 
accuracy with values of 56%,76%, and 77% respectively, indicating acceptable 
preliminary results. Collecting more data and exploring the parameters of Tab- 
Net could potentially improve its performance. In addition to its performance, 
XGBoost exhibited faster training and validation times compared to TabNet. 

We compare our results to a previous study [14] in Table 5. In [14], the 
authors developed an MLP classifier to detect the challenging behaviors of an 
ASD male child using four statistical features (min, max, mean, and std) from 
EDA, BVP, HR, and ACC. Table 5 compares the results achieved using XGBoost 
and the performance of the MLP in [14]. 


Table 5. Performance comparison of our model and the results of [14] 


Metrics Our Method | MLP [14] 
Precision 0.77 0.73 
Recall 0.60 0.53 
F 1-score 0.67 0.61 
AUC 0.80 0.76 
PR AUC 0.72 0.67 
Balanced Accuracy | 0.80 0.76 


This comparative Table 5 shows that our method achieved higher perfor- 
mance. In fact, the features extracted in our paper contain a large amount of 
information, which could reflect the behavioral state of the child. 
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Fig. 1. The importance of features determined by XGBoost classifier of each signal 
EDA (a), BVP (b), and ACC (c) 


With a focus to identify the contribution of each signal to our detection 
model, we calculated the features importance of the XGBoost classifier. The 
feature vector combines EDA, BVP, and ACC information respectively resulting 
in a 179—dimensional vector. Figure 1 displays the contribution of each feature 
of each signal: EDA la, BVP 1b, and ACC Ic. 

Based on our results, the extracted features of the EDA, BVP, and ACC 
signal contributed 80.15%, 5.6%, and 14.25% respectively to XGBoost. These 
results confirm the literature assumption about the performance of EDA in 
identifying the psychophysiological state of individuals. As mentioned in Sect. 2, 
EDA has been extensively used in studies related to arousal investigation such 
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as stress and anxiety, and emotion detection. Conversely, BVP contributes less 
to the detection of agitated behaviors, potentially due to the fact that this signal 
is very susceptible to environmental artifacts, leading to a less reliable heart rate 
and heart rate variability readings. 

Previous research focused on kinetic data to detect agitated behaviors. How- 
ever, this kind of signal solely could not contribute to a good generalization. Our 
results demonstrated the feasibility of using physiological biomarkers combined 
with kinetic data to detect agitated behaviors of people on the spectrum. 


5 Conclusion 


Agitated behaviors represent significant challenges for individuals with autism 
and their caregivers, impacting their well-being. Technological solutions could 
improve their quality of life. This paper attempted to study the viability of using 
signals collected from wearable devices during everyday activities to detect agita- 
tion. The proposed framework involved pre-processing to remove additive noise, 
followed by relevant features extraction for each type of signal. The features were 
later combined to train different classifiers to detect agitated behaviors. 

Our approach showed promising results across multiple evaluation metrics. 
More specifically, XGBoost achieved the highest performance in terms of bal- 
anced accuracy, precision, recall, F1l-score, AUC, and PR-AUC. 

These findings suffer from some limitations including a limited number of 
recorded moments of agitation. Moreover, we were only able to recruit a sin- 
gle ASD child. We firmly believe that the proposed models could be improved 
through the collection of a larger dataset from a larger number of ASD patients, 
hence refining the performance of classifiers. 

The implication of our findings is significant in terms of enhancing the qual- 
ity of life of patients with ASD. By providing caregivers with timely notification 
of moments of agitation, they can prevent harmful consequences associated with 
these destructive behaviors. Our proposed framework holds the potential to alle- 
viate the negative impact of these behaviors and promote a safer and more sup- 
portive environment for patients with autism. Additionally, our future work will 
focus as well on developing predictive models capable of anticipating agitated 
behaviors before their occurrence. This proactive approach in this case will help 
caregivers to intervene early and prevent unwanted consequences. 
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Abstract. The development of light-based technologies has facilitated the use of 
phototherapy devices in the medical field. Representative phototherapy devices 
are laser surgical instruments and medical laser irradiators. While the former are 
used for the removal, ablation, and destruction of areas in need of treatment using 
the light of a strong laser; the latter irradiate the skin, induce skin regeneration, and 
relieve pain [1]. These medical devices are used under the guidance of healthcare 
providers in medical institutions and can be hazardous because they use strong 
light. According to the Korean Ministry of Food and Drug Safety, medical devices 
are divided into four classes according to the hazard level [2]. Devices such as 
lasers are currently represented by Class 3, which represents a high risk. However, 
various personal phototherapy devices have recently emerged and are used without 
any instructions from healthcare providers. Among them, representative personal 
phototherapy devices include those that use LED. Such devices are known to have 
effects such as pain and inflammation relief, scar healing, and skin regeneration 
according to previous studies [3]. With the progress of research, LED medical 
devices for skin improvement and beauty treatment are emerging in Korea. Thus, 
LED medical devices are used for various purposes. Certain devices only use 
LEDs, while others are used in com-bination with lasers their boundary with 
medical laser irradiators is vague. Based on the report on personal phototherapy 
devices by the Korea Consumer Agency, there were 172 adverse events of LED 
masks, which were home phototherapy devices, from 2018 to Au-gust 2020; they 
comprise 134 cases of skin and subcutaneous tissue injuries, six of burns, one of 
heat sensation and dyspnoea, and one of bruising [4]. As various phototherapy 
de-vices continue to be developed, assuring adequate safety is paramount. This 
study con-tributes to the industrial development of LED phototherapy devices and 
new phototherapy devices. We derived a performance test method to ensure safety 
related to the performance of LED phototherapy devices in the Korean context. 
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1 Introduction 


The development of light-based technologies has facilitated the use of various types of 
phototherapy devices in the medical field. Representative phototherapy devices are laser 
surgical instruments and medical laser irradiators. The former are used for the removal, 
ablation, and destruction of areas in need of treatment using the light of a strong laser; 
the latter irradiate the skin, induce skin regeneration, and relieve pain [1]. These medical 
devices are used under the guidance of healthcare providers in medical institutions and 
can be hazardous because they use strong light. According to the Korean Ministry of Food 
and Drug Safety, medical devices are divided into four classes according to the hazard 
level [2]. Devices such as lasers are currently represented by Class 3, which is the grade 
with high risk. However, various types of personal phototherapy devices have recently 
emerged and are used without any instructions from healthcare providers. Among them, 
representative personal phototherapy devices include phototherapy devices using LED. 
LED phototherapy devices use a low-level phototherapeutic method known as Low- 
Level Laser Therapy (LLLT). LLLT is used for relief of pain or inflammation, scar 
healing, and skin resurfacing using low output power [3]. LED phototherapy devices 
with LLLT are used for skin improvement and cosmetic purposes. They vary in shape 
depending on the site at which to be used, and are of various types. Certain devices 
use only LEDs, while others are used in combination with laser—their boundary with 
medical laser irradiators is vague. Based on the report on personal phototherapy devices 
by the Korea Consumer Agency, there were 172 adverse events of LED masks, which 
were home phototherapy devices, from 2018 to August 2020. They comprise 134 cases 
of skin and subcutaneous tissue injuries, 6 of burns, 1 of heat sensation and dyspnoea, 
and 1 of bruise [4]. As various phototherapy devices continue to be developed, assuring 
adequate safety is paramount. 

This study contributed to the industrial development of LED phototherapy devices 
and new phototherapy devices. We derived a performance test method to secure 
safety related to the performance of phototherapy devices using LEDs, from a Korean 
perspective. 


2 Materials and Method 


2.1 Determination of Performance Test Items Related to LED Phototherapy 
Devices 


It’s determine to derive the performance test method to preserve the stability of LED 
optical medical devices and to obtain suitable test materials for these devices by inves- 
tigating performance issues and conducting related performance tests. As a research 
method, we proceed by reviewing previous research and standards related to optical 
medical devices such as International Organization for Standardization (ISO) and Inter- 
national Electrotechnical Commission (IEC). For the literature review, a literature search 
medium was used, and the search was conducted with “LLLT,’ “PBM,” and “LED” as 
search keywords. Based on the reviewed literature, performance test items and methods 
were derived by examining key performance issues related to optical medical device 
therapy and LED optical medical device standards. 
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2.2 Performance Test Method for the Phototherapy Device Sample 


We examined the safety of the LED phototherapy devices by applying the performance 
test items derived from this study to the LED medical device developed by T company, 
shown in Fig. 1. The company’s LED phototherapy device samples were developed for 
the treatment of hair loss and scalp improvement and was shaped like a helmet. The 
LED phototherapy device comprises 54 lasers and 126 tri-color LEDs. Its performance 
items are 10 mW for the output power of the laser, 3 mW for the intended output power, 
and 650 nm for wavelength. Each LED has an output power of 3 mW, with wavelengths 
of red 625 nm, blue 455 nm, and infrared (IR) 845 nm. The key performance test items 
and methods of optical medical devices are obtained and used in the model to protect 
the validity of the test. 


Fig. 1. LED phototherapy device sample. 


3 Results 


3.1 LED Optical Medical Device Literature Investigation 


As aresult of the literature review, three volumes were reviewed for LLLT, two volumes 
for LED, two volumes for PBM optical medical device-related literature, and five vol- 
umes for related standards. In addition, two books related to Korean medical devices and 
photo-therapy were mentioned. By reviewing the research literature, main performance 
items according to the treatment of LED light medical devices, standards related to LED 
light medical devices, and performance test items and methods were obtained. 


3.1.1 Key Performance Points for Treatment Using LED Optical Medical Devices 


Phototherapy is a medical technique that has been used for thousands of years to treat 
various diseases. An example is sun therapy by Hippocrates, the father of Western 
medicine [5]. In modern times, various treatments using light have evolved. A common 
treatment that uses LEDs is photobiomodulation (PBM). This was previously called 
Low-Level Laser Therapy (LLLT) and was later changed to Low-Level Light Therapy 
by Kendrick C. Smith due to the advent of the next generation LED with higher photon 
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intensity. Photobiologist [6]. After the change, the name was again changed to PBM due 
to uncertainty about the exact meaning of low power [7]. PBM was proposed by Andre 
Meister (1903-1984), a medical expert who researched and published the biological 
effects of low-power lasers in 1967 [8]. According to the literature, PBM is a treatment 
that uses a low-power laser or LED in the 1m to 50mW range to induce tissue regeneration 
and relieve inflammation. Red or near-infrared light with a wavelength of 600 nm to 
1000 nm is used, and, unlike other laser treatments that destroy tissue, PBM is known for 
its photochemical effect similar to plant photosynthesis [9]. The mechanism of PBM is 
not yet clearly elucidated; however, when tissue is irradiated with low-output IR light, itis 
absorbed by a chromophore that penetrates the skin surface and absorbs light in a specific 
wavelength band. In the body, at this time [10], the absorbed light energy is converted into 
chemical energy and increases the synthesis of Adenosine Triphosphate (ATP), which 
is the energy source of the cells in the cell tissue, resulting in cell regeneration and 
rehabilitation, pain and inflammation relief, which is known [11, 12]. According to the 
literature related to LED-LLLT, the main performance issues for LED light sources used 
in PPM include wavelength, power density, and luminance. In terms of wavelength, 
the first law of light biology, the Grothes-Draper law, states that when the light of 
the appropriate wavelength is applied to the chromophore, the chromophore has no 
absorption nor reaction. Power density refers to the output used per unit area, and the 
unit is W/Cm^2. When the power density is insufficient, light absorption does not occur 
in the tissue, and when the power density is high, it is converted to excessive heat for the 
tissue, which could be harmful. Fluence (energy density) or “dose” refers to the amount 
of energy applied per unit area. The energy density is multiplied by the radiation time, 
and the unit is J/Cm^2. If the power density is too low, the irradiation time must be 
extended to achieve the best power density, which is why the desired results cannot be 
obtained [6, 10]. 


3.1.2 Items for Standards Related to LED Phototherapy Devices 


To derive LED phototherapy device test items, the relevant standards were investigated 
by referring to the international standards, ISO and IEC, as well as Korean standards. 
Currently, there are two main categories of common standards for medical devices in 
South Korea. The classification consists of standards applied comprehensively and stan- 
dards applied separately by medical device items. Additionally, it comprises medical 
devices that use electricity as a whole and those that do not use electricity [13]. Compre- 
hensive standards are common standards for electromechanical safety, electro-magnetic 
waves and safety, and biological safety. The common standards for electromechanical 
safety are those applicable to the basic safety and essential performance of medical 
electrical appliances and medical electrical systems [13]. In the common standards on 
electromechanical safety, standards related to phototherapy devices were IEC 60601-1, 
IEC 60601-2-22, IEC 60825-1, IEC 60825-8, and IEC 62471-2. IEC 60601-1 is a set 
of standards that ensure the safety of medical electrical equipment, cover basic safety 
and essential performance requirements of medical electrical equipment, and prevent 
electrical, mechanical, or functional errors that present unacceptable hazards to patients 
and operators [14]. IEC 60601-2-22 is an individual standard of IEC 60601-1 standard 
applied to basic safety and essential performance of surgical, therapeutic, diagnostic, 
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cosmetic, and animal laser instruments for use in humans or animals. IEC 60601-2-22 
is a standard applicable to phototherapy devices corresponding to classes 3B to 4 and 
applied to the safety of laser output power. For example, on the 201.12.1.101 laser out- 
put power display, the actual laser output power measured in the operating area should 
not exceed +20% of the set value [15]. IEC 60825-1 is a standard applied to the safety 
of laser products that emit light with a wavelength range of 180 nm to | mm. Per this 
standard, classes are di-vided into 1, 1M, 1C, 2, 2M, 3R, 3B, and 4, as described in 
Table | [16]. 


Table 1. Laser classes according to IEC 60825-1 [16]. 


Classes | Explanation Wavelength | Output 
(mW) 

1 It is not dangerous to direct the laser beam at the human body | Visible - 

1M It is dangerous to see or direct laser beams with optical devices Invisible 


(However, products with access emission limits of Class 1) 


IC Products that directly use lasers for treatment or diagnosis of 
the skin or internal tissue. Eye exposure can be controlled by 
engineering methods, even if the emitted laser is rated 3R, 3B, 
and 4th 


2 When laser beams are irradiated in the eyes, they can be Visible <1 
protected with a blink of 0.25 s 
(However, products with access emission limits of 0.25 s or 
less in pulse width are rated 1.) 


2M It is dangerous if laser beams are directly used with optical 
devices 
(However, the restriction on access emission is a Class 2 
product.) 

3R It is dangerous if laser beams are directly used on the eyes Visible 1~5 

Invisible 

3B It is dangerous if laser beams are directly used on the human | Visible 5 ~ 500 
body Invisible 

4 It is dangerous if laser beams are directly used on the human | Visible >500 
body Invisible 


These ratings are classified as per the accessible emission limit (AEL) according to 
the Maximum Permissible Exposure (MPE) in Annex A of IEC 60825-1, as shown in 
Table 2. The MPE provided in IEC 60825-1 presents additional information that can 
assist the manufacturer in assessing safety items for the intended use of the user. The 
MPE contained in the standard has been adopted from exposure limits published by the 
International Commission on Non-Ionized Radiation Protection, and MPE values have 
been used as the basis for safe design of products and for providing information to users. 
[16] IEC 60825-8 provides guidance to employers, competent authorities, laser safety 
managers, and others, on the safe use of laser and laser equipment classified as 3B and 4. 
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An investigation identified potential side effects according to wavelengths in Appendix 
A of IEC 60825-1, which are shown in Table 2 [17]. 


3.1.3 LED Phototherapy Device Performance Test Items and Method Derivation 


The main performance requirements and specifications related to performance test-ing 
for optical medical devices were grasped through the literature. It has been con-firmed 
that wavelength, power density, and fluorescence are key performance parameters for 
LED optical medical devices. If a wavelength that does not match the chromophore 
of the target to be treated is used, suitable results cannot be obtained; in the case of 
fluorescence that does not match, the desired result cannot be obtained, and thermal 
damage can be expected. Therefore here, the key performance test items obtained were 
wavelength, power, and power density, as shown in Table 3. For assessing wavelength, 
we used the wave-length accuracy test method according to IEC 62471-1 Annex B; 
the accurate calculation formula was used to obtain the measured value and the actual 
value as a percent-age, and the quality of the test + L1% within the error range was 
determined. Power density is the output per unit area, and an accuracy test for output 
and a test for illumination was obtained. Output Accuracy Test IEC 62471-1 Annex C. 
As atest method, the measured and the actual values are calculated as a percentage using 
a power meter. The quality of the test should be + LJ% with an error margin of + LI%. 
The test for illumination is IEC 62471-1. 5, wherein we follow the measurements of the 
lamp and the lamp device and measure it using an illuminance meter. The test standard is 
IEC 62471-1. 4. 4. The test method that follows the hazard exposure limits is IEC 62471- 
1.5. The test method was obtained ac-cording to the measurement of lamps and lighting 
fixtures, and IEC 62471-1 4. Exposure limit standards were followed. Energy density, 
one of the key performance factors, is the product of energy density and irradiation time, 
and the test item of energy density was obtained. However, irradiation time was excluded 
because it was difficult to be obtained. The test item will vary depending on the LED 
optical medical device. 


3.2 LED Optical Medical Device Literature Investigation 


Among the test items obtained from Sect. 3.1.3, the wavelength accuracy test was applied 
to LEDs built into the Company T model, and the same test was performed on the laser 
built into the model. To measure the wavelength, as shown in Fig. 2 below, we want to 
measure the main performance factors of the two models of the T company, the laser 
wavelength, and the LED wavelength, using the spectrometer of the T company. The 
two models have the same performance and we wanted to compare them by measuring 
the wavelengths in each mode. 

As for the measurement method, the light source to be measured was aligned with the 
spectrometer’s wavelength meter sensor, as shown in Fig. 3. The wavelength meas-ured 
was then checked by the monitoring function in the software on the computer con-nected 
to the spectrometer. 
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Table 2. Laser classes according to IEC 60825-1 [16]. 


Test item 


Wavelength 
accuracy test 


Test standard 


Within the 
error range + 
% 


Test method 


Measure the wavelength of the LED light source using the 
photodetector. Check whether the measured wavelength value 
is suitable in terms of the test standard after calculating by 
applying the measured and the actual values to the equation 
below: 

Accuracy (% ) = 

[[(Actual value — Measured value) ]]/[[Actual value]] x 100 


Output Within the Measure the output of the LED light source using a power 
Accuracy error range + | meter. After calculation, substitute the measured output value 
Test % and the actual value in the formula below to check if it meets 

the test standards 

Accuracy (% ) = 

[[(Actual value — Measured value)]]/[[Actual value]] x 100 
Illuminance | IEC In accordance with IEC 62471-1 5. Measurement of lamps 
measurement | 62471-1.4. and lamp devices. Measure the illuminance of the LED light 

Follow the source using a luminance meter 


exposure limit 
standards for 
harm 


Table 3. Measured values by mode in the samples of T company’s device and the accuracy of 


wavelength. 

Wavelength LASER IR LED RED LED BLUE LED 

Set value: 650 nm Set value: 845 nm Set value: 625 nm Set value: 455 nm 
Mode Sample 1 | Sample 2 | Sample 1 | Sample 2 | Sample 1 | Sample 2 | Sample | Sample 2 

1 

H Mode 654 nm 652 nm 838 nm 840 nm 627 nm 627 nm - - 
S Mode 654 nm 653 nm 838 nm 839 nm - - 455 nm | 454nm 
Accuracy of | H +0.615% | +0.31% | +0.828% | 40.59% |+40.32% | 40.32% |- - 
wavelength |g +£0.615% | +0.461% | +0.828% | +0.71% |- = +0% | 0.219% 


3.3 Performance Test Results of LED Phototherapy Devices 


The two samples of T company were numbered 1 and 2, and the laser and LED were 
measured for each mode of the samples. The set values for each mode of samples were 
the same for the laser and IR LED, and the set values for the laser and the IR LED were 
650 nm and 845 nm, respectively. The set values for the red LED in H (Hair) mode and 
the blue LED in S (Scalp) mode were 625 nm and 455 nm, respectively. The measured 
values for the two samples are shown in Fig. 4. 

The laser wavelengths per the mode of T company’s samples measured with the 
spectrometer are shown in Fig. 5. The graph shows a maximum peak value of 654 nm. 
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Fig. 2. Wavelength performance test devices (Left: a spectrometer, Right: the sensor of the 
spectrometer) 


Fig. 3. Wavelength measurement for the light source. 
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Fig. 4. Results of laser wavelength measurements by mode of T company’s samples. 


When the measured wavelength band was visually checked, the graphs overlapped be- 
cause the measured laser wavelength bands were similar to the set value of 650 nm. 
However, as it was difficult to accurately differentiate the superimposed graph with the 
naked eye, the graph was adjusted to distinguish the laser wavelength band measured, 
as shown in Fig. 6. Using the adjusted graph, the measured wavelength for each mode 
of the two samples and the accuracy of the laser wavelengths were checked. 

Laser measurements of T company’s samples revealed 654 nm in H mode, 654 nm in 
S mode (Sample-1), 652 nm in H mode, and 653 nm in S mode (Sample-2). They differed 
by 2 to 4 nm from the set value of 650 nm in the laser. As each of the measured lasers was 
the same for each of the samples, Sample 1 indicated the same value. However, Sample 2, 
though it was the same object, demonstrated a difference of 1 nm depending on the mode. 
When each measured value was applied to the wavelength accuracy equation, Sample 
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Fig. 5. Enhanced graph of laser wavelengths measured in modes of T company’s samples. 
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Fig. 6. LED wavelength graph measured in sample modes of T company’s device. 


1 showed an error range of +0.615% because the values per mode were the same. As 
for Sample 2, the error range was +0.31% in H mode and +0.461% in S mode. The 
laser wave-lengths of the two samples were within the error range of +5%, which was 
the test standard set by the manufacturer. For both samples, LED measurements with 
the spectrometer revealed the maximum peak values of 627 nm, 454 nm, and 838 nm, 
which were visible to the eye. When the graph was checked, both samples were found 
to be similarly matched and overlapped with the laser graph measured earlier. However, 
a difference of 1 to 3 nm was observed compared to the LED set values of 625 nm for 
red, 455 nm for blue, and 845 nm for IR. The graphs were adjusted, as shown in Figs. 7, 
8, and 9 for precise determination. The respective graphs confirmed LED wavelengths 
by each mode and the accuracy of wave-lengths for the two samples. 

The IR LED measured value was 838 nm in both H and S modes for Sample 1. In 
Sample 2, the values for H and S modes were 840 nm and 839 nm, respectively. For each 
wavelength, there were differences of 7 nm in Sample 1 and 5 to 6 nm in Sample 2 from 
the set value of 845 nm. The wavelength accuracy of Sample 1 was within +0.828% 
because all measured values were the same. The wavelength accuracy of Sample 2 was 
+0.591% in H mode and +0.71% in S mode. Both samples were within the test standard 
of +5%. In the case of IR LED and laser, the same object was applied for each mode. 
The measured values according to the mode were distinctly different from the set value. 
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Fig. 7. Graph of laser wavelengths. 
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Fig. 8. Red LED wavelength graph measured in the samples of T company’s device. 


The red LED of T company’s samples was used only in H mod 


e. The measured 


values in Samples 1 and 2 were 627 nm with a difference of 2 nm from the set value 
of 625 nm. The accuracy of the red wavelength was +0.32% for both samples, thus 


indicating that both samples had an error range within +5%. Unlike 


the laser and IR 


LED measured earlier, the red LEDs had the same values for Samples 1 and 2, and the 


difference from the set value was small. 
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Fig. 9. Blue LED wavelength graph measured in the samples of T comp 


The blue LED of T company’s samples is used only in S mode. The 
were 455 nm in Sample 1 and 454 nm in Sample 2. In Sample 1, the 


any’s device. 


measured values 
measured value 


was the same as the set value of 455 nm, whereas Sample 2 had a difference of 1 nm. 


Regarding the accuracy of wavelength, Sample 1 had an error range of 4 


0% because the 
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measured value was equal to the set value, whereas Sample 2 had +0.219% in the error 
range. Thus, the values were within the test standard of +5% in the error range. In this 
study, we tested the wavelength performance of both the laser and the LED used in the 
samples of the T company, from which we found that all measured values were within 
the reference values. The measured value of the blue LED was the closest to the set value 
than the data measured earlier, which was found stable, as shown in Fig. 9. The values 
of the wave-lengths measured in this study and their accuracy are shown in Table 4. In 
conclusion, all measured values were within +5%, the test standard. However, the laser 
and IR LED, which are used regardless of mode, had larger differences between the 
measured values and the set values among the measurements. 


4 Conclusions 


This study investigated items related to the performance of LED phototherapy devices to 
ensure their safety. Although LLLT using LED phototherapy devices, which have been 
recently examined, have no reported principle, it was possible to identify wavelength 
as the main performance item, which is the same for other phototherapy devices. The 
standard of phototherapy devices equipped with both LED and lasers such as T com- 
pany’s phototherapy device needs to be explored in combination with LED and laser 
standards. The investigation of standards was able to identify the classes of medical laser 
de-vices in IEC 60825-1 and their hazard depending on wavelength in IEC 60825-8, the 
appendix of IEC 60825-1. However, certain laser standards were found to be applied to 
3B or higher classes of a laser. As the samples of T company’s device used in this study 
were used in combination with LED and low-level laser, it was difficult to investigate 
the standards because of the differentiation between LED and laser. For LED photother- 
apy devices, IEC 62471-1, which is the same standard as lamp-based medical devices, 
was applied. Referring to IEC 62471-1 and the relevant literature and guidelines, this 
study de-rived the performance test method and test items for wavelength, the main 
performance item, and applied them to samples of the T company’s device. The derived 
performance evaluation item was the accuracy test of the wavelength according to IEC 
62471-1. The test was to determine whether the LED and laser wavelengths were stable. 
The test standard was within +5% in the error range corresponding to the test standard of 
T company. The test method checked whether the measured wavelength shown through 
the spectrometer was within the error range through the percentage. According to the 
measurement results, the light sources of the samples complied with the test standard set 
by the manufacturer. The measured wavelength for samples differed from the set value 
by 1 to 2 nm in red and blue LEDs, and the accuracy error range of the wavelength was 
close to 0%. However, laser and IR LEDs, unlike red and blue LEDs, had a difference 
as large as 2 to 7 nm between the set value and the measured value, though they are used 
regardless of the mode. More-over, the error range was also shown to be close to 1%, 
unlike those of the red and blue LEDs. To make it closer to the set value, it is necessary to 
study the wave-length of photo-therapy devices whose accuracy is important for safety. 
We believe that a study on the wavelength of the phototherapy device similar to the set 
value would con-tribute to the safety of LED phototherapy devices and the industrial 
advancement of phototherapy de-vices similar to LED phototherapy devices. 
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Abstract. Falls are a significant health risk for older adults, and fall- 
related injuries are a leading cause of morbidity and mortality in this pop- 
ulation. Elderlies are particularly vulnerable to falls due to age-related 
declines in mobility, balance, and muscle strength, as well as chronic 
medical conditions with use of certain medications. These injuries can 
range from minor bruises and scrapes to more severe like fractures, head 
trauma, or internal bleeding. To prevent falls in older adults, some solu- 
tions propose to ensure a safe living environment, others to maintain 
physical activity, and others to manage chronic medical conditions. This 
article presents the implementation and test of a system preventing hip 
fractures resulting from falls using a fall detection and prediction system 
designed to protect and alert individuals during falls. 


Keywords: Pre-impact fall detection - fall detection - machine 
learning - tinyML - medical devices + IMU - e-health - airbag system 


1 Introduction 


Falls among the elderly are currently a real human and financial scourge. 25 to 
30% of people over 65 and 50% of those over 80 fall at least once a year. These 
are the first cause of emergency visits for people over 75 and the second cause of 
death by unintentional trauma. A significant proportion of falls result in femoral 
neck fractures, a real public health issue (Katsoulis, 2017 [1] Oberlin, 2016 [2]). 
Furthermore, research has demonstrated that the apprehension of falling is linked 
to unfavorable outcomes, including reduced engagement in everyday activities, 
decreased level of physical activity, increased likelihood of falling, symptoms of 
depression, and a lower overall quality of life [3]. According to INSERM (2017) 
[4], between 50,000 and 80,000 individuals in France, and approximately 1.5 
million around the world (Silver Eco 2017 portal) experience fractures of the 
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neck of the femur. This type of fracture has severe implications as one-quarter 
of people pass away within one year, and 50% of those who survive experience a 
considerable loss of autonomy. This work is part of a project led by Indienov, a 
company that created a smart airbag belt capable of analyzing the movements 
of individuals and detecting anomalies (in this particular case falls and pre- 
falls), and instantly activate a perfectly positioned airbag prevent fractures of 
the femoral neck or greater trochanter (Fig. 1). The belt will also be able to 
detect critical situations such as lying for a prolonged time. 


2 Literature Review 


A large variety of fall detection systems are available and rely on diverse sensors 
(cameras, infrared motion sensors, LIDAR, and inertial motion units). These 
fall detection systems are generally classified into two categories: fixed systems 
(in the environment) and mobile/wearable systems (necklaces, watches, belts, 
phones, etc.) [5]. We focused on mobile systems using Inertial Motion Units. 


2.1 State of the Art 


Initial research concerning fall detection systems employed threshold-based algo- 
rithms, and demonstrated that by applying thresholds to vertical and horizontal 
velocities, it is possible to differentiate between daily activities and falls [6,7] 
with remarkable accuracy. In a study based on a sensor located on the chest 
[20], a fall detection algorithm is optimized using the Upper Fall Threshold 
(UFT) and Lower Fall Threshold (LFT) of the accelerometer combined with 
the UFT of the gyroscope for fall detection. The experimental results showed 
that the algorithm was able to detect falls compared to other daily movements 
with a sensitivity and specificity of 96.3% and 96.2%, respectively. The addi- 
tion of gyroscope data significantly improves specificity over published results 
based solely on accelerometer data, as changes in angular velocity provide an 
additional indication of a fall event compared to other activities that may also 
experience high acceleration spikes. The use of LFTs and UFTs is equivalent to 
determining the window of fall and detecting it by means of its start and end. 
Alternatively, Li et al. [8] used static postures and dynamic transitions 
between them to detect falls, defining a fall as a static “lying” posture with an 
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unintentional transition to it. Numerous other studies have examined fall detec- 
tion, including reviews, learning algorithms with automatic feature extraction, 
etc. [9-18], some of which have been found to be highly reliable [19]. 

Detecting the onset of a fall, or pre-fall detection, presents a challenging 
problem. Unlike fall detection, which relies on identifying the impact of a fall 
visible on acceleration signals, pre-fall detection requires detecting the early signs 
of an impending fall. As a result, accelerometer and gyroscope signals may not 
yet exhibit features indicative of a fall. The primary hurdle in solving this issue 
is to distinguish the crucial warning signs that trigger a fall while minimizing 
the number of inaccurate detections (false positive cases). 

In the context of pre-fall detection, one of the earliest fall protection systems 
was proposed by G. Shi et al. [21] and later enhanced [22]. It relies on an iner- 
tial motion unit and airbag deployment, along with a Support Vector Machine 
(SVM) prediction model and an embedded Digital Signal Processing (DSP) unit. 

In recent years, machine learning models have gained popularity due to their 
good performances in fall and pre-fall detection. In a comparative study con- 
ducted by Yu et al. [24], a hybrid ConvLSTM model was evaluated and com- 
pared to other existing models. The results showed that the ConvLSTM model 
outperformed other models for all three classes (no drop, drop before impact, and 
drop), with average sensitivities of 93.15%, 93.78%, and 96.00% for no drop, drop 
before impact, and drop, respectively. Furthermore, the model demonstrated 
higher specificities for all three classes (96.59%, 94.49%, and 98.69%) compared 
to the LSTM and CNN models. These results suggest that the proposed hybrid 
model outperforms the LSTM and CNN models, providing high detection accu- 
racy (particularly for the pre-impact drop). The combination of convolution and 
recurrent models allows feature extraction and temporal linkage between the 
data, leading to satisfactory results compared to using these models separately. 

Recent fall prediction methods have shown very promising results (as shown 
in Table 1). However, one important issue highlighted in the literature is the need 
to compare the performance of fall detection algorithms using real-world data, 


Table 1. Fall prediction algorithms comparison 


Fall prediction algorithms overview 


Réf Sensor | Placement | Sampling [Hz] | Methods FALL/ADL | Subjects | The best results 
[23] ae etal A trunk 100 HMM 2/5 8 Sp : 100% 
TI : 200-400 [ms] 
, Sn : 92.26 % 
[25] cn e al ALG | back-trunk |20 NN N/A 5 Sp : 70.02 % 
TI : 400 [ms] 
TOR - Sn : 97.8% 
[26] S et al. | 256 ANE MobiFall| Sp : 99.1% 
; TI : 300—700[ms] 
, x Sn : 93.78% 
[24] Yuet al. | / 200 CNN TSTM SisFall | Sp : 94.00% 
(2020) ConvLstm 
Tl: / [ms] 
os Sn : 99.32% 
Ki Gen al A, G, M back-waist | 100 Pe mr am 21/35 32 Sp : 99.01% 
2 OMSL Tl: 403 + 163 [ms] 
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Table 2. Comparison of the Kfall model performances on simulated data and real 
data. Source : [27] 


Algorithm Evaluation on real word data 


Réf Algorithm | Data FN (false negative) | FP (false positive) | Sensitivity (%) | Specificity (%) | Lead time ([ms]) 
E al: | ConvLSTM | KFall dataset | 3/444 5/507 99.32 99.01 403 + 163 
Farseeing dataset | 1/15 4/15 93.33 73.33 411 + 317 
8 


as significant discrepancies have been observed between the results obtained 
from simulated falls data of healthy individuals and real falls data from at-risk 
populations [11]. The Bourke and Chen algorithms perform the best in terms of 
sensitivity and specificity, but all algorithms exhibit lower performance on real 
falls compared to simulated falls. The latter being also true for the work from 
Yu et al. (as shown in Table 2) despite the very high performance obtained on 
simulation data (99.32%, 99.03% of sensitivity, and specificity, respectively). This 
difference in performance was mainly attributed to the fact that the movements 
in real falls do not exhibit the same signatures as simulated falls, especially in 
terms of the amplitude before impact [28]. 


3 Methods and Material 


3.1 Data Collection 


As mentioned above, the problem of motion and fall data for at-risk individuals 
currently remains one of the biggest challenges. The first step consists of obtain- 
ing real, reliable motion data that comes from our target audience. Then, in 
addition to retrieving widely used movement data from the internet (especially 
in research, e.g. SisFall [30], MobiFall [29], kFall [27] etc.), a specific protocol 
based on [29,30] was developed to gather experimental data (Table 3 and 4). 

A data collection campaign has been initiated in EHPAD (institutions for 
dependent elderly people) (Dataset_03, Dataset_04), in which multiple volun- 
teers were equipped with dataloggers (embedded sensors with IMU + SD card - 
attached to a belt worn at the waist) to monitor their activities throughout the 
day (and in some cases, the entire week from morning to evening). The purpose 
of the database obtained from this data collection is to develop solutions and 
algorithms adapted to the target population, the elderly, where detecting pre-fall 
events is of utmost importance. As the participants’ profiles are highly suitable 


Table 3. Public Data Overview 


Public Dataset 


Dataset | Labelled | Types ADLs/Falls Number of participants | Age Weight [KG] | gender (M/F) 
MobiFall | Yes 9/4 24 2247 | / 17/7 

23 19-30 11/12 

SisFall | Yes 19/15 15 60-75 41-102 8/7 

kFall Yes 21/15 32 24 + 3.7 | 69.3 + 9.5 32/ 
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Table 4. Acquired Data Overview 


Recolted Data 


Dataset Labelled | Types ADLs/Falls | Number of participants | Age | Weight [KG] | gender (M/F) 
Dataset_01 | Yes 11/8 1 47 / H 

Dataset_02 | Yes 47/24 11 22-52 | 50-120 4/7 
Dataset_03 | Yes 13/0 ~ 60 >63 | 47-120 ~ 38/22 
Dataset_04 | No / > 100 >63 | 47-130 >40/>60 


for this purpose, the collected data is expected to provide valuable insights and 
aid in validating the developed solutions and algorithms. 


3.2 Configuration and Setup 


After examining the SisFall and MobiFall datasets, we discovered that they were 
not optimal for predicting falls because they lacked temporal information on the 
pre-fall phase. Even though there has been a lot of research on fall detection, 
there hasn’t been much done on prefall detection. The key contrast between 
prefall detection and fall detection is found here. Prefall detection focuses on 
detecting the beginning of a fall, which happens over a brief period of time and 
then passes. In other words, throughout the fall, we can only see the beginning of 
the fall during a specified time period. On the other side, fall detection focuses on 
identifying a fall when it really occurs, generally after the impact has taken place. 
At any time after the fall, it is relatively simple to identify because the impact 
of the fall clearly leaves an imprint in the signals. Therefore, we had to identify 
and label the pre-fall phase, which may have introduced bias into the labeling 
process. In contrast, the kfall dataset contained more detailed information on the 
onset and impact of falls, making it more useful for fall prediction. Additionally, 
the age and profiles of the participants in the various datasets were not very 
representative, so we generated new data for activities of daily living and falls 
that included a wide range of realistic scenarios. 

For the labeled data, participants followed a predefined protocol that included 
activities from the SisFall dataset as well as additional daily activities such 
as walking, using stairs with a cane or walker, doing household chores, and 
dancing. We also included wheelchair activities. Unlabeled data were collected 
from participants engaged in their normal activities throughout the day, resulting 
in data that better reflects real-world scenarios. For data recording, we used 
dataloggers, which are developed by the same company, and we recorded the 
videos of these simulations. The IMU (MPU6050) that we used was equipped 
with a 3-axis accelerometer and gyroscope, with a resolution of +8 [g] for the 
accelerometer and +1000°/s for the gyroscope. The sampling frequency (200 
[Hz]) is defined based on the data analysis and signal processing. 
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3.3 Data Processing and Feature Extraction 


Data Cleaning and Labelling: To reduce accelerometer sensitivity noise, we 
used a 3rd order Butterworth lowpass filter with a cutoff frequency of 20 [Hz] 
as first step. We then divided the fall data into four stages (Fig. 2): non-risk, 
pre-fall, the impact, and the post-fall. This categorization was based on notes 
taken during fall simulations (such as the type of fall, subject, and duration) 
and recorded video data (including the start time of the fall, the end time of the 
pre-fall phase, the time of ground contact, and the time of impact). 


Data Analysis and Feature Selection: An initial visual inspection was 
performed to analyze the collected data. This enabled the identification of 
movement-specific signatures and variable behavior, and the tracking of their 
fluctuations over time in relation to distinct activities and subject profiles (as 
shown in Fig.3). A statistical examination of the information obtained from 
various activities was performed with the aim of identifying indicators that can 


Acceleration [g] 


m oor 


6 
Time [s] 


Pre-fall 


Fall impact 


Fall duration : ~750[ms] 


Fig. 2. Fall Data Segmentation. 


Fig. 3. Activities signal examples - Acc: accelerometer (acc_x, acc_y, acc_z), Gyro: 
gyroscope (gyro_x, gyro_y, gyro-z) - 3 Normal Activities (1, 2, 3) vs one Fall Activity 
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Fig. 4. Analysis of the data distribution examples - compare the data distribution over 
different activities (ADLs vs FALLs) 


Table 5. Selected Variables for Feature calculation 


Selected Variables for Features calculation 


Variable description 
Acc_x The acceleration data over the x-axis 
Acc_y The acceleration data over the y-axis 
Acc.z The acceleration data over the z-axis 
Gyro_x The angular velocity data over the x-axis 
Gyro_y The angular velocity data over the y-axis 
Gyro_z The angular velocity data over the z-axis 


The magnitude of the acceleration vector (1) 
Acc_svm = acc x? + acc_y? + acc_z?. 
The square root of the sum of the angular velocities squared 
Gyro_svm = ygyrox? + gyro_y? + gyro_z?. 
The measure of the magnitude of a varying acceleration 


Acc_svm(signal vector magnitude) 


Gyro_-svm 


(2) 


SMA( signal magnitude area 
( ) + Je (ace_x(t) = Hase-2 F lacc_y(t) = Bazel F |acc_z(t) = figassa] 


; The vertical tilt of the body 
Tilt tilt = arctan( aoe J; (4) 


a/ acc_x? +acc_z2 


yat. © 


distinguish between ADLs and falls (Fig. 4). Once this stage was completed, we 
selected the variables that would be used in computing the features (listed in 
Table 5). Subsequently, statistical computations were carried out on each vari- 
able, resulting in a collection of potential features for pre-fall detection (Table 6). 
The features were calculated in sliding windows of 0.5{s]. 


Algorithm and Features: For feature selection, we started an optimization 
process over all the features (cited on Table 6), based on recursive elimination of 
features by KNN (K Nearest Neighbors) and RF (Random Forest) algorithms, so 
we tested several combinations depending on the type of falls we were proposing 
and the performances we were getting. Ultimately, we chose an algorithm based 
on the RF (random forest) and the most important features, which we selected 
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Table 6. Selected Features for Calculation 


Selected Features for calculation 
Feature description 
mean the mean value of the current window 
max The max value of the current window 
min The min value of the current window 
std(Standard Deviation) | The standard deviation value of the current window 


mean-start the mean of the m first values ([0; m]) of the current window 


the mean of the m values around the median 


({median_index - (%4); median_index + (4)]) of the current window. 


mean_median 


mean_ql the mean of the m values around the 1st quartile of the current window. 
mean_q3 the mean of the m values around the 3rd quartile of the current window. 
diff_m1_m2 the difference between each two calculated means, cited above 


measures the asymmetry of the distribution of the current window 

skewness EP (x-y)? (5) 
skewness = =a — 

measures the “tailedness” of the probability distribution of the current window 

ET Gi-n)4 (6) 


mo4 


kurtosis 5 
kurtosis = 
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Fig. 5. Fall Detection/Prediction System. 


for their relevance to decision making. Our solution detects pre-falls and falls 
(Fig. 5), then, sends an alert to relatives and caregivers (previously selected). 


3.4 Model Deployment in Embedded Architecture 


Due to their inherent resource constraints, embedded systems are unable to 
support sophisticated inference models. As a result, implementing AI models 
in these systems becomes difficult, especially in real-time scenarios when char- 
acteristics like response speed, space complexity, and computing complexity are 
critical. Focusing on low complexity choices is necessary since these limits impose 
extra factors to take into account when choosing the right model. Deep learning 
techniques, for instance, may not be practical due to their inference, time, and 
space complexity, necessitating careful attention throughout implementation. As 
a result, it may be necessary to develop customized methods to optimize model 
training, conversion, and implementation for better optimization and utiliza- 
tion of hardware resources such as DSP and FPU. Various methods (weight 
pruning, quantization, encoding, etc.) and solutions (TensorFlow Lite Micro, 
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EdgeML, STM32Cube.AI, etc.) are available to address this challenge [31], but 
they require careful consideration and planning to achieve the desired results in 
the context of TinyML. 


4 Experiments and Evaluation 


4.1 Experiments 


Initially, we tested our solution on daily life and fall acquired data to ensure its 
functionality (this was the Experiment_00). Our ultimate goal was to develop 
a functional and customized solution for the airbag belt. Therefore, we imple- 
mented and deployed the solution in the embedded system (belt) and conducted 
three types of experiments. 


— Experiment_00: In this experiment, we have tested our solution on the 
acquired data (which contains ADLs and Falls). 

— Experiment_01: The first test involved wearing the belt and performing 
guided activities (ADLs and falls). 

— Experiment_02: The second test involved wearing the belt throughout the 
day and engaging in their daily activities. 

— Experiment_03: Finally, we tested the belt on elderly people in EHPAD for 
a whole day and evaluated the performance recorded in the log files. 


The first and second tests were conducted by young and healthy individuals. 


4.2 Evaluation 


We established a set of evaluation criteria for our solutions, which serve to mea- 
sure their quality and degree of reliability. These criteria include: 
. Sensitivity: measures the solution’s ability to detect and predict falls accu- 
rately. Sensitivity = TP LEN? TP : true positive, FN : false negative. (7) 
. Specificity: measures its ability to correctly identify activities of daily living 
(ADLs). Specificity = TFTP: TN : true negative, FP : false positive. (8) 
. Time Lead: which refers to the time between the prediction of a fall and 
the actual impact of the fall. Our objective is to detect pre-fall movements at 
least 220 [ms] before impact (safe deployment of airbags). 


Finally, we use performance as an overall measure of how well our solution 
performs based on the aforementioned criteria. This value is calculated using 
a weighted sum of sensitivity, specificity, and time lead. Based on the desired 
outcomes, it is possible to set the weights of the parameters, such as emphasizing 
sensitivity, specificity, or lead time. 


Performances = (Psn * Sensivity) + (Psp * Specifity) + (Pim * timelead). (9) 
Performances € [0,1], P: weights, P; € [0, 1], >». P; = 1, time_lead normalized 
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Table 7. Experimental results 


Experimental results 


/ PDS (pre-fall detection system) FDS (fall detection system) 

/ Sensitivity(%) | Specificity(%) | Lead time[ms] | Sensitivity(%) | Specificity(%) 
Experiment _00 | 99 98 ~ 250 99 99 
Experiment_01 | 88 92 | ~ 210 98 | 97 
Experiment_02 | / 87 | / / | 97 
Experiment_03 | / 93 | / / | 97 


5 Experimental Result Analysis 


The Dataset_02 was used for training, and subsequent optimization techniques 
were implemented to select a subset of the data that would yield lightweight 
and tailored models optimized for embedded systems. To assess the effectiveness 
of our solution, we employed different methods depending on the experiment. 
For Exp. 0 and Exp. 1, we evaluated the performance based on the number of 
simulations (each activity labeled as ADL or Fall), enabling us to determine false 
positive and false negative rates. For other experiments (exp. 2 and exp. 3), we 
based our evaluations on the number of activity hours and participant-reported 
information. If the participant doesn’t fall but a fall was predicted or detected, 
it is considered a false positive. If the participant falls, but we didn’t detect 
anything, it is considered a false negative. However, if the person falls and we 
predict it, it was considered as a true positive. 

Generally, falls in real life are infrequent, so our solution specificity is high- 
lighted in exp. 2 and exp. 3. When we encounter false positives or false negatives, 
we analyze the data and incorporate it into subsequent training sessions. The 
benefit of these experiments is that we can evaluate and validate our solution 
under real-world conditions with participants who are natural and free, unlike 
other evaluation tests. Therefore, our solution is exposed to situations (activi- 
ties) not seen during training, representing a challenge for us. The evaluation 
results that we currently have are presented in (Table 7). It should be noted that 
the evaluation of our solution is still ongoing, with particular focus on (Exp. 03). 


6 Discussion 


Based on the initial results from experiments (exp. 00 and exp. 01), it appears 
that fall detection is facilitated by the fact that signals already contain the neces- 
sary information to accurately detect falls. This is supported by the high level of 
sensitivity and specificity observed. The fall prediction models showed a decrease 
in performance when the environment and subjects were changed (between exp. 
00 and exp. 01), which is likely attributed to the prior compression of the mod- 
els before deployment on embedded devices. This compression was intended to 
reduce the models space-time complexity and make them more lightweight. The 
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two additional experiments exposed our solution to real-world conditions, which 
allowed participants greater freedom of movement, resulting in signals that dif- 
fered, sometimes markedly, from what we had observed in our learning database. 
Moreover, as the belt-wearing was subtle, participants behaved more naturally, 
revealing movements we had not encountered during training. 


7 Conclusion 


This study involved the development of a fall detection system and a fall pre- 
diction system, which were adapted and deployed on an airbag belt to test their 
effectiveness in real-life conditions. While the theoretical results were comparable 
to other works, the goal was to create a functional system that performed well 
in both theory and practice. This is in contrast to some studies that relied on 
theoretical results or private motion data to which the researchers did not have 
access. However, the lack of movement data from elderly people and real fall 
data posed significant challenges. Therefore, the plan for the future is to launch 
a collection campaign in EHPAD to gather natural data over a long period of 
time, including real falls that can be added to the learning base. 
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Abstract. Remote patient monitoring (RPM) has been gaining popularity 
recently. However, health data acquisition is a significant challenge associated 
with patient monitoring. In continuous RPM, health data acquisition may miss 
health data during transmission. Missing data compromises the quality and reli- 
ability of patient risk assessment. Several studies suggested techniques for ana- 
lyzing missing data; however, many are unsuitable for RPM. These techniques 
neglect the variability of missing data and provide biased results with imputation. 
Therefore, a holistic approach must consider the correlation and variability of the 
various vitals and avoid biased imputation. This paper proposes a coherent com- 
putation pattern-matching technique to identify and predict missing data patterns. 
The performance of the proposed approach is evaluated using data collected from 
a field trial. Results show that the technique can effectively identify and predict 
missing patterns. 


Keywords: Missing health data - Pattern-matching technique - Remote patient 
monitoring - loMT - Smart healthcare 


1 Introduction 


In recent years, wireless vital sign sensors, commonly known as the Internet of Medical 
Things (IoMT), have broadly extended the boundaries of remote patient monitoring 
(RPM) [1, 2]. These sensors remotely and continuously measure a patient’s five vital 
signs, such as oxygen saturation (SpO2), blood pressure (BP), body temperature (Temp), 
respiratory rate (RR), and heart rate (HR), which allows for early identification of any 
abnormalities or deterioration in the patient’s conditions [3]. Continuously monitored 
patient data and its extracted features can be used for decision-making, early clinical 
event prediction models [4], and automated risk analysis [5, 6]. 

The modified early warning score (MEWS) is widely used in hospital wards for 
nurses to assess a patient’s condition and raise alerts [7]. First introduced as an early warn- 
ing score (EWS) [8], it was replaced with the modified early warning score (MEWS) [9, 
10]. Weighted scores from 0 to 3 for each five vital signs parameter datasets are assigned 
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from thresholds on raw values; single vital parameter scores are summed to generate a 
MEWS score [10, 11], as shown in Table |. Several variations to patient deterioration 
using MEWS and other approaches, including single parameter, multiple parameters, 
the aggregate score, and combination system methodologies, have been reviewed [11]; 
however, approaches commonly assume the minimal value of the parameter if there 
is no value for a particular parameter. Therefore, missing data refers to any parameter 
value with no recorded data in the five vital signs parameter dataset., Much research 
has focused on dealing with missing data [12, 13]; however, missing data in continu- 
ous vital signs in RPM is an unexplored area. The number of parameter data observed 
continuously in RPM makes the database highly susceptible to missing data occurrence, 
and its analysis is a significant challenge. Missing data is generally categorized into 
three categories: Missing completely at random (MCAR), Missing at random (MAR), 
Missing not at random (MNAR) [13]. 


Table 1. Modified Early Warning Score [9, 10] 


Vital Signs 3 2 1 0 1 2 3 
Systolic Blood Pressure(BP) | <70 71-80 81-100 | 101-199 | — >=200 |- 
(mmHg) 

Heart-Rate (HR)(bpm) - <40 | 41-50 |51-100 101-110 | 111-129 | >=130 
Respiratory Rate(RR) (bpm) | — <9 — 9-14 15-20 21-29 >=30 
Temperature (Temp) (degree | — <35 |- 35-38.4 |- >=38.5 |- 

C) 

Oxygen Saturation (SpO2) | <85 | 85-89 90-93 | >94 - - - 


The amount of data missing in any clinical setting depends upon various causes, such 
as delayed data transmission, environmental issues, connectivity issues due to a network, 
sensor malfunction, power loss, and sensor detachment from the skin. Consequently, 
missing or poor-quality data will impact the evaluation of vital sign abnormalities, 
clinical event detection algorithms, or the risk analysis of the patient [14]. 

Extensive work exists on analyzing missing data in the literature [13, 15]. A gen- 
eral regression neural network (GRNN) and successive geometric transformation model 
(SGTM) were applied in a study to handle the missing clinical data [16]. Missing data 
may frequently occur in RPM monitoring, and the amount of recorded data variations 
makes it challenging to analyze as these algorithms need significant data for training 
[17]. In addition, these techniques often ignore unique patterns in data and correlations 
between the observed vitals. 
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This study proposes a similar pattern match to continuously provide missing values 
imputation based on a temporal association of vitals in a sliding window approach. 
Patterned early modified warning score (PMEWS) is introduced in the research as a 
pre-processing feature for finding pattern matches for the missing vital signs values. 
This pattern-matching technique helps identify the trends and patterns in the data even 
when the variability of the missing data is significant. This approach uses the required 
vital signs data from a clinical trial from our RPM real-time implementation [18]. The 
main contribution of this study is summarized below: 


e We propose a pattern-matching algorithm to impute the missing sensor data from 
different sensor streams. Our algorithm creates the patterns of vital signs values 
appearing at a particular time to impute missing data using the pre-processing feature 
PMEWS. 

e Our novel algorithm can detect and predict similar matching patterns for the missing 
values. 

e The proposed approach utilizes the MAR and MCAR to simulate incomplete data 
proportions of 10%, 20%, and 30%. Then, to check the accuracy of our proposed 
algorithm, the actual dataset without omission is verified with our suggested matching 
of similar patterns for the missing values. 


This paper is organized as follows: Sect. 2 outlines the related work of this research. 
Section 3 describes our proposed solution. Section 4 describes our results, and Sect. 5 
concludes the paper. 


2 Related Work 


Imputation is the recovery-based approach to overcome the limitation of complete case 
analysis if the missing data falls under the MCAR category [14]. Imputation is used to 
replace the missing data values using single mean imputation, conditional mean impu- 
tation, and last observation carries forward (LOCF), multiple imputations (MI), and 
full information maximum likelihood (FIML) [19]. Single-mean imputation, where the 
missing data is imputed by the mean of the recorded value of that attribute, has a lim- 
itation that affects the dataset’s variation. In comparison, conditional mean imputation 
intensifies the multivariate relationship in the data. Due to these limitations of mean and 
conditional imputation, these are recommended for baseline measurements in random- 
ized trials [20]. LOCF is similar to single imputation, which replaces the missing values 
with the last observed value. MI has three phases: the imputation phase, the analysis 
phase, and the pooling phase [21]. MI generates n replacements for missing values. 
Then these completed datasets generated with n imputed values are analyzed and inte- 
grated into the final output. MI applies to both MAR and MCAR. FIML best suits the 
MAR context as it considers only the observed data and ignores non-response data [22]. 
These imputation methods are inappropriate if the missing data falls under the MNAR 
category. 
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Random Forest is inherently a MI approach, and it finds the mean of the data from 
numerous unpruned classification or regression trees for missForest [23, 24]. Bayesian 
ridge regression used the probability distributor when designing the linear regression, 
which enables the automated process for the missing data [25]. A cluster’s variable 
mean, or mode is used to impute the missing values in the hot/cold deck [26]. Using the 
Euclidean distance, the K-nearest neighbour imputes the missing values with a similar 
one by defining the similarity between two values [26]. 

Normalization, formatting, and synchronization of the data are also necessary [27]. 
Normalizing the recorded data variables is essential as various scoring systems, such as 
the sequential organ failure assessment (SOFA) and MEWS, were registered at different 
frequencies [28]. Removing abnormal values from the data is also described in the 
literature as a part of the pre-processing [29]. One typical statistical analysis, the complete 
case, removes all the rows with the missing values and studies all accounts with the 
exclusive data [30]. This method has a disadvantage: it will reduce the amount of valuable 
data, producing biased results. Von Russum [12] analyzes techniques such as linear 
interpolation, spline interpolation, last observation, mean-forward, and cluster-based 
imputation for missing values. Imputation using these resulted in early warning score 
(EWS) miscalculations ranging from 1% to 8%. These methods produced more biased 
results than oversized observation windows [12]. A deep learning-based protocol for 
accurately predicting the missing data is also proposed in the literature [31]. Expectation 
Maximization (EM) [32] is a popular algorithm for imputing the missing data. One 
disadvantage of EM is that the recorded data should be highly correlative to get more 
reliable information [33]. 

Existing approaches perform well in imputation accuracy, though their efficiency 
is greatly affected by various factors such as computational time, complexity, and cor- 
relation of parameters. Our proposed pattern-matching imputation approach considers 
factors such as the correlation of the parameters, which helps to reduce the biased impu- 
tation and computational time can be set using the sliding window to show efficient 
performance in handling the missing values. 


3 Pattern-Matching for Missing Values 


The structure of our method is divided into two stages: Initially, the data is pre-processed 
using the patterns created using the MEWS [34] discussed in Sect. 3.1. Pre-processing 
the data helps ensure an accurate assessment of the patient’s condition. Pre-processing 
includes removing erroneous values, no recorded vital values at a time, abnormal values, 
and high-frequency noise [35]. In the second stage, a pattern match is found for the 
missing values in an observation window. These two components work together to 
provide a high similarity match for the missing data. 
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Vital signs data for parameters systolic blood pressure, blood oxygen, body tem- 
perature, respiratory rate, and heart rate streams at a discrete time for each parameter 
using the wearable sensors recorded from patients in a general ward [18] were used in 
the current study. Publicly available datasets were not used to ensure data is simulated 
real data and were not curated in any other ways. The imputation method considers the 
data’s temporal context and more accurately predicts the missing values using a sliding 
window. A sliding window has been used to find patterns for a particular data segment 
[21, 22]. A six-hour prediction window for clinical deterioration is considered ideal [36]. 
The sliding window is predefined in our algorithm. 

A sliding window analyses a subset of data within the process window of fixed length, 
often with a minimum overlap of one data point and shifting with the window increment 
time in the process window. The notations used for the pattern-match algorithm are the 
number of vitals to be recorded (Ty), Sliding window time (t), String patterned array 
(Sp[i]), Variables (Str1, Str2), Counter for the matched character (CSp[k]). Figure 1 
shows the sliding window increment and overlaps in the process window. T10, T20, 
T30, T40, T50, and T60 denote the time in Fig. 1. The sliding window (sw5) consists of 


different time slots y +n) shown in Eq. (1). 


Sliding Window 


Window Increment 
(5) 


Fig. 1. Sliding window with the overlap and window increment 


The process window length (L), where (L e N) N is the set of all the natural numbers, 
and the slide window increment (8), where (1 < = 5 < L), can be decided by the physi- 
cian, as shown in Fig. 1. If urgent attention is needed, a shorter window increment time 
can deliver more frequent information on the patient’s condition. On the other hand, a 
longer window increment can be employed to lower the computation burden and boost 
the analysis accuracy if the patient’s condition is stable and not urgent. A significant 
observation window with personalized decision-making and imputation produced pos- 
itive results for missing values on maternal health data [14]. The window number (i) 
where (i € N). Notation (tis a) will be written as TS for calculations in the rest of the 
paper, as shown in Eq. (2). 


SWS = (Estoy tat), IGH creer te+n)) (1) 


TS = (t5 4n) (2) 
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The physician will decide the start time (T) and the time slot (TS) for the observation. 
The slide window increment (8) value will be valid for the time slot (TS) and can be 
changed for the next process window (L). “Patterned modified early warning score” 
(PMEWS) is described in the next section. The PMEWS forms the basis for similar 
pattern matches for missing vitals values. The method is described in the next section 
after PMEWS. 


3.1 Pattern Modified Early Warning Score (PMEWS) 


Our method for handling missing values in our data set is to make a pattern of the 
values from the recorded set, replacing the missing values with letters or symbols. As 
shown in Table 2, recorded raw, vital values at any data points are converted into their 
respective MEWS score to form the pattern, where ‘N’ is used for the null values. 
Another significant benefit of using the pattern with the letters is that it avoids using any 
previously registered threshold value that could affect the prediction of the patient’s risk 
assessment by providing a biased estimation. In our method, patterns can be recorded in 
any format. 

An ordered sequence of MEWS values that make a pattern is recorded from the 
patient’s observed data and is labelled as the PMEWS. The pattern array best suits sensor 
signal discrepancies because of null values and discrete-time readings from different 
sensors. This step is the second most vital step in raw data transformation. 


Table 2. Pattern Formation 


HR BP SpO2 RR Temp Pattern 
Raw-vital values 99 — 96 22 — 
MEWS 0 N 0 2 N ONO2N 


For sliding windows (swi) consisting of time slots (TS), threshold data values of 
patients’ vitals are recorded after processing them into their relevant MEWS score (M), 
and the resulting pattern is stored in an integer array P[TS]. Equation (3) shows an 
example of the pattern at the time slot (TS). MHR, MBP, MT, MSPO2, and MRR are the 
MEWS scores derived from the raw, vital data for heart rate, blood pressure, temperature, 
oxygen saturation and respiratory rate, respectively. 


PITS] = [MHR, MBP, MT, MSPO2, MRR] (3) 


The following section describes the pattern-match algorithm to identify similar 
patterns for missing values of vital signs. 
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3.2 Pattern-Matching Algorithm 


Our main aim for this algorithm is to match the pattern with the missing value with the 
closest matching pattern that appeared in the sliding window. When the match is found, 
imputing the missing value parameter with the found match values is performed. For 
example, in Table 3, the raw data is recorded for the heart rate (HR), temperature (T), 
blood pressure (BP), oxygen saturation (SpO2), and respiratory rate (RR) and relevant 
patterns are created for the time one minute to four minutes. HR is missing at minute 
four with the pattern “NNN20’ and will get pattern-match at minute one ‘ONN20’ as the 
highest number of symbols matched and imputed with the HR value of 99. 


Table 3. Pattern Match 


Time HR T BP RR SpO2 Pattern Symbol match 
(minutes) 

1 99 - - 22 96 ONN20 4 

2 97 - 141 25 99 ON020 

3 102 - - 34 99 INN30 2 

4 = - - 28 99 NNN20 


After pre-processing, the first step is to initialize the variables T, L, t, TMv, 8, swi, 
and TS. The pattern P[TS] is generated from MEWS for the recorded raw values. Loop 
to traverse the string pattern Sp[], from the start of a sliding window, i = T to TS is set. 
If there is a pattern Sp[i + 1] with no null values, the counter i is incremented for the 
following index. Otherwise, another inner loop is set for the string pattern array with the 
index j = T toi to ensure all patterns are considered to match null data values. To ensure 
a match, two variables, strl and str2, are set with SP[j] and SP[i + 1], respectively. 
A call to the function count passes these two string variables to check if strl contains 
str2. For a matching pattern character case, the count is maintained for the pattern and 
stored in counter array CSp[k]. The counter array CSp[k] from j = T to TS is checked 
for the condition if the number of character matches is highest for the number of vitals 
observed, as shown in Table 4. Relative patterns and count values are displayed as 
suggestions for outcomes. Timeslot TS is updated with the sliding window increment 
time 8 for traversing the sliding window loop. This algorithm is an iterative process in 
the sliding window to ensure all patterns are considered. 
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The proposed algorithm that accomplishes our pattern matching is shown as follows: 


Start 

Input 

Process window length (L), start time (T), 
to be recorded (Tw), window increment (6), 
time (t), Where: 

sliding window sws = (tis+0) €(s41,l(642) e e e e Leen) 
time slot  TS= (forny) 

While (TS < = L) do 


number of vitals 
sliding window 


Record values of vitals (HR, Temp, BP, RR, and SpOə2) 

// Assign MEWS to vitals, where null assigns "N" 

Save P[TS] = [MHR, MBP, MT, MSPO2, MRR] 
Copy recorded patterns in Sp[ ] 
for each pattern P[TS] in Spl 


] from i=T to i<=TS do 
if Sp[ i+1] != `N’ then i++_ 
else assign: 
for (int j=T; j<=i; j++) do 
stri = Sp[ j ] 
str2 = Sp[ itl] 
count (stri; str2) 


for each count c for pattern in array CSp[ k ] 
from j = T to TS do 
CSp[k]= c , end for 
for each count in CSp[k] from j = T to TS do 
If (CSp[k] is the highest match parameter 
count) then 
print relevant pattern P[TS] and count from 
CSp[k] 
Else print no suitable match found 
end if, end for 
end if, end for 
Increment TS = t + 8 , end While 
function count (string strl, string str2) 


initialize counter c = 0, index j = 0 
for each i in strl.length() do 
If 
(str2.find(strl1[i])>=0 (AND) j==strl.find(strl1[i]) 
) then 
ct=1, jt=1 
return c, 


end if, end for 
end function, Stop 
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4 Result and Discussion 


To assess the plausibility of our technique, we used the dataset collected from the clinical 
trial [18]. The trial dataset has the raw values for the vital parameters Temp, BP, SpO2, 
RR, HR, and the time for each record, as shown in Table 3. For this study, only two 
parameters, SpO2 and HR, were imputed. The dataset was split into additional sets of 
various record lengths. MI and EM are commonly used to impute missing values; our 
proposed method is compared against these. Normalization is an essential step in pre- 
processing our data, as it helps scale the data and make it more suitable for analysis by 
reducing the biased outcome. Normalization is performed using the MEWS. Starting 
with the entire collection of values, we removed various percentages of the SpO2 and 
HR to test the precision of our technique. The values of the missing parameters in the 
dataset were then imputed using our pattern-match imputation, EM and MI in the same 
sliding window. The highest match for parameters is selected for the imputation by our 
algorithm. For comparison, we calculated the root mean squared error (RMSE) for the 
imputed value with the starting values from the whole dataset. Table 4 presents results 
obtained from four different datasets using different imputation methods. 
For dataset D1: 


e SpOy: Pattern-match has the lowest RMSE of 0.82, outperforming EM and MI. 
e HR: Pattern-match has the lowest RMSE of 3.9, outperforming EM and MI. 


For dataset D2: 


e SpOz: Pattern-match has the lowest RMSE of 2.03, outperforming EM and MI. 
e HR: Pattern-match has an RMSE of 13.9, higher than EM and MI. 


For dataset D3: 


e SpOz: Pattern-match has an RMSE of 2.4, while EM has the lowest. 
e HR: Pattern-match has the lowest RMSE of 3.9, followed by EM and MI. 


For dataset D4: 


e SpOz2: Pattern-match has an RMSE of 2.5, while EM has the lowest. 
e HR: Pattern-match has the lowest RMSE of 7.3, followed by EM and MI. 


Figures 2 and 3 show the variation of RMSE of two of our used parameters, SpO2 
and heart rate (HR). In the context of missing data, the sliding window approach and 
the PMEWS have proven very useful as the algorithm demonstrates comparable results. 
The algorithm considers every pattern in the sliding window before the missing pattern 
and uses these patterns to find a similar match. Then it slides the window across the 
process window to cover all the patterns that appeared in that timeframe. 
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Table 4. Results of the Datasets. 
Dataset | Total Records | Parameter observed | Observed Records | Method RMSE 
D1 756 SpO2 45 Pattern-Match | 0.82 
EM 2.01 
MI 2.52 
D1 756 HR Zi, Pattern-Match | 3.9 
EM 9.6 
MI 10.2 
D2 1000 SpO2 88 Pattern-Match | 2.03 
EM 2.1 
MI 2.94 
D2 1000 HR 26 Pattern-Match | 13.9 
EM 20.6 
MI 24.3 
D3 490 SpO2 25 Pattern-Match | 2.4 
EM 1.2 
MI 2.04 
D3 490 HR 20 Pattern-Match | 3.9 
EM 4.3 
MI 7.4 
D4 932 SpO2 84 Pattern-Match | 2.5 
EM 1.39 
MI 2.06 
D4 932 HR 28 Pattern-Match 7.3 
EM 7.8 
MI 14.04 


The proposed algorithm’s accuracy was satisfactory, which suggested that this app- 
roach is practical for finding a similar effective match for the missing vital parameter 
values in medical datasets generated in RPM settings. It was observed that most of the 
similar patterns in data appeared after the normalization phase of pre-processing of the 
data. The algorithm can leverage these patterns to find identical matches for the miss- 
ing vital values, which can improve the accuracy of the prediction. These findings have 
implications for the researchers, physicians, and practitioners who work with the vital 
sign data set with the missing values in RPM for patient risk assessment analysis. 
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5 Conclusion 


This study presents a method for pattern-matching the missing values in the RPM vital 
sign dataset using the PMEWS and the sliding window. This method offers the identi- 
fication of similar patterns in the data, and using the PMEWS, this technique is further 
improved by leveraging these patterns. Compared to other approaches, this technique 
considers the temporal context of the missing values, which can be particularly impor- 
tant in the medical domain where the timings of the measurements can have critical 
implications for patient care. This method also has the potential to revolutionize how 
continuous RPM data is processed and analyzed, leading to more efficient healthcare 
delivery. Future research could explore other pre-processing techniques to improve the 
pattern match further. Overall, the proposed method can improve the accuracy of the 
medical data analysis for decision-making, specifically for the missing vitals values. 
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Abstract. Arm support is a typical assistive technology device to assist shoulder 
and elbow movements in those with reduced upper extremity muscle strength due 
to neurological lesions. Recently, the assisting method, range, and manipulation 
method of arm support are also changing with the development of technology. 
Accordingly, an assessment system is required for appropriate matching and mea- 
suring the effectiveness in the clinical field. This study examines the direction of 
the assessment process by analyzing studies that measured the effectiveness of arm 
support through a systematic literature review. The databases were collected using 
Embase, CINAHL Plus with Full text, Web of Science, and Scopus. 19 studies were 
selected according to the Preferred Reporting Items for Systematic Reviews and 
Meta-Analysis (PRISMA) flow chart. Most studies have been conducted on people 
with neuromuscular disorders. Assessment area included performance, function, 
usability, satisfaction, and psychosocial impact. And more than half of the studies 
measured performance and function. There were various assessment methods to 
measure the effectiveness including assessment tools, kinematics, physical exam- 
ination, questionnaire, observation, and EMG, with assessment tools accounting 
for more than half of the studies. Most studies have set up assessment environments 
based on tasks related to ADLs and IADLs to measure the effectiveness. Currently, 
various methods such as assessment tools and kinematics were applied to measure 
the effectiveness of arm support. The assessment tool was used the most among 
them. However, assistive technology-based assessment tools are extremely lim- 
ited. Therefore, it is required to develop an assessment tool centered on assistive 
technology based on performance related to ADL and IADL in the future. 


Keywords: Arm support - Assessment - Systematic review 


1 Introduction 


Arm support is an assistive technology device (ATDs) that has been used since the 
1960s to supplement the performance of activities of daily living (ADLs) of those who 
(e.g., Spinal cord injury, Muscular dystrophy, Multiple sclerosis) limited upper extremity 
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muscle strength [1]. It provides stability to the proximal part of the shoulder by removing 
gravity acting on the upper limb through an armrest and assists shoulder and elbow 
movement through a power source [2]. In recent, the power source of arm support is 
developing with the development of technology. Because of this, people with disabilities 
who use arm support can receive support for a wider range of movements. According to 
a systematic review by Lee et al. (2021), the proportion of arm supports was the highest 
among the upper extremity ATDs to assist people with disabilities in their ADLs [3]. 
Also, they found that the power source for assisting the user’s movement is changing from 
a spring to an actuator, electricity to assist the user’s active movement according to the 
development of technology. Changes in power sources will affect how users manipulate 
arm support and the extent to which it can assist a user’s functional movements [4]. 
Therefore, in the clinical field, it is required to set up an assessment system that can 
prescribe appropriate arm support according to the function of the people with disabilities 
and measure the effectiveness after the intervention. Standardized assessment about 
ATDs relies on the subjective judgment of the user, such as satisfaction and psychosocial 
impact [5, 6]. In addition, objective assessment systems and methods for performance 
are limited. This is a critical barrier to assessing the multifaceted effectiveness of ATDs. 
Therefore, the purpose of this study is to collect and systematically analyze studies that 
have been assessed about arm support, and to suggest directions for establishing an 
assessment system. 


2 Method 


2.1 Database and Search Term 


This study searched for studies that applied arm support as an intervention in 4 literature 
search databases (Embase, CINAHL Plus with Full text, Web of science, and Scopus). 
Search terms were “Arm support” AND (Assessment OR Evaluation OR Measurement) 
and search period was set to the past 10 years. 


2.2 Selection Strategy 


The studies were selected according to the Preferred Reporting Items for Systematic 
Reviews and Meta-Analysis (PRISMA) statement [7]. 

Studies were included if they met the following criteria: (1) studies on people with 
disabilities, (2) studies that assessed after providing arm support as an intervention, 
(3) can see the full text, (4) studies written in English or Korean. And, studies were 
excluded if they met the following criteria: (1) studies that not provided arm support as 
an intervention, (2) studies about reliability and validity of assessment tool, (3) studies on 
people without disabilities, (4) Systematic review & meta-analysis, (5) Grey literature, 
(6) Studies that could not open the full text. 

Literature selection was carried out by two researchers. In case of disagreement 
between the researchers, it was discussed and agreed upon. Consequently, 19 studies 
were selected for this study (Fig. 1). 
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Fig. 1. PRISMA flow chart 


2.3 Analysis 


The selected articles were analyzed frequency for the type of disability, assessment area, 
assessment method, frequency of assessment tools by area, and measurement method of 
assessment tools. 


3 Result 


3.1 Type of Disability 


In terms of the type of disability, most of the studies have been conducted on people who 
have decreased performance in daily activities mainly due to decreased muscle strength 
in the shoulders and elbows. Sixteen studies were conducted on people with muscular 
dystrophy which is a loss of strength from the proximal to the distal part of the shoulder. 
It is occupied the largest proportion. Eight studies were conducted on people with spinal 
muscular atrophy. And three studies were conducted on people with spinal cord injury, 
stroke, and arthrogryposis, respectively (Table 1). 
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Table 1. Type of disabilities on subjects 


Type of Disability N (%) 
Muscular Dystrophy (Duchenne, Becker, Facioscapulohumeral, etc.) 16(33.9) 
Spinal Muscular Atrophy 8(17.0) 
Spinal Cord Injury 3(6.4) 
Stroke 3(6.4) 
Arthrogryposis 3(6.4) 
Cerebral Palsy 2(4.3) 
Amyotrophic Lateral Sclerosis 2(4.3) 
Multiple Sclerosis 2(4.3) 
Etc 8(17.0) 
Total 47(100.0) 


3.2 Assessment Area 


In the included studies, Upper extremity performance and function were primarily 
assessed about arm support interventions. This is consistent with the results of previous 
research [3]. It can be confirmed that it is an important factor to measure the function 
and ADL performance in the assessment of arm support intervention. In addition, it 
was found that assessment was performed through the subjective judgment on usability, 
satisfaction, psychosocial influence, fatigue, and workload according to the use of the 
arm support (Table 2). 


Table 2. Assessment area included studies 


Area N (%) 
Performance 14(45.2) 
Function 10(32.3) 
Usability 3(9.7) 
Fatigue 1(3.2) 
Satisfaction 1(3.2) 
Workload 1(3.2) 
Psychosocial impact 1(3.2) 
Total 31(100.0) 
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3.3 Assessment Method 


The included studies used either quantitative or qualitative method to assess the effect 
of arm support on the area. Most studies used standardized assessment tool to measure 
effectiveness of arm support. By using a standardized assessment tool, it is possible to 
secure the validity and reliability of the results. Kinematics through 3D motion analysis 
was used to measure performance and function based on tasks (e.g., Workspace, Range 
of motion). Physical examination which is the traditional method of measuring function 
mainly assessed range of motion (ROM) with a goniometer and manual muscle strength 
(MMT). Through qualitative methods such as questionnaires and observations, factors 
that are difficult to measure quantitatively were assessed (Table 3). 


Table 3. Assessment Method included studies 


Method N (%) 
Assessment tool 18(50.0) 
Kinematics 6(16.7) 
Physical examination 411.1) 
Questionnaire 4(11.1) 
Observation 3(8.3) 
Electromyography 1(2.8) 
Total 36(100.0) 


3.4 Assessment Tool by Area 


Looking at the assessment tools by area, tools for measuring performance and function 
occupied about 80% of the total. Since there is no assessment tool centered on assistive 
technology, the occupational therapy assessment tool was used or modified for assess 
on performance and function. There were two assistive technology-focused assessment 
tools: QUEST which measures satisfaction with the device and service process, and 
PIADS which measures the psychosocial impact of using assistive technology (Table 4). 
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Table 4. Assessment tool by area included studies 


Area Assessment tool N (%) 
Performance Performance of the Upper Limb (PUL) 5(18.5) 
Abilhand Questionnaire 3(11.2) 
Canadian Occupational Performance Measure (COPM) | 2(7.4) 
TEMPA 1(3.7) 
Upper-Limb Functional Index (ULFI) 1(3.7) 
Function Box and Block Test 2(7.4) 
9-Hole Peg Test 2(7.4) 
Motor Assessment Scale (MAS) 1(3.7) 
Action Research Arm Test (ARAT) 1(3.7) 
Motor Function Measure (MFM) 1(3.7) 
Jebsen Taylor Hand Function Test 1(3.7) 
Alberta Infant Motor Scale (AIMS) 1(3.7) 
Usability System Usability Scale (SUS) 2(7.4) 
Fatigue OMNI-RPE 1(3.7) 
Satisfaction Quebec User Evaluation of Satisfaction with assistive 1(3.7) 
Technology (QUEST) 
Psychosocial impact Psychosocial Impact of Assistive Devices Scale (PIADS) | 1(3.7) 
Workload NASA-TLX 1(3.7) 
Total 27(100) 


3.5 Measurement Method of Assessment Tool 


Most assessments measure effectiveness through the performance of tasks related to 
ADL, which are mostly utilized to measure performance and function. A questionnaire- 
based assessment tool is used to investigate users’ subjective feelings about the effec- 
tiveness of arm support. There were also studies that used semi-structured interview and 
observation to measure the effectiveness of the arm support (Table 5). 


Table 5. Measurement method of assessment tool included studies 


Measurement method N (%) 
Performance 14(51.9) 
Questionnaire 10(37.0) 
Interview 2(7.4) 
Observation 13.7) 
Total 27(100.0) 
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4 Conclusion 


Arm support is an assistive device that compensates for shoulder and elbow movements 
to help those with reduced upper extremity muscle strength perform activities ADLs. 
Due to the development of technology, the assisting method and range, and operation 
method of arm support are also changing. Accordingly, a system for properly matching 
and assessing effects in consideration of the user’s function, environment, and activity is 
required in the clinical field. In this study, we examined assessment trends and suggest an 
appropriate direction for assessment by systematic review on studies that measured the 
effectiveness of arm support. First, the assessment area for arm support is mainly focused 
on performance and function. The use of assistive devices interacts not only with the 
user but also with the activity and environment. Therefore, a comprehensive assessment 
considering not only users but also activities and environments should be conducted. 
Second, assessment tools are the most popular way to measure the effectiveness of arm 
support. However, when looking at the trend of tools by assessment area, the assessment 
tools centered on assistive technology are extremely limited. The majority of studies 
use modifications of existing occupational therapy assessments to measure performance 
and function. It is necessary to develop an assistive technology-centered performance 
assessment tool or method. Third, the effectiveness of arm support is measured through 
task-based performance. Assigned tasks are mainly composed of ADLs and instrumental 
activities of daily living ADLs). Therefore, it is required to establish an environment 
for measuring effects based on ADLs and IADLs related to using arm support. We expect 
this to be useful in the process of measuring the effect after providing arm support as an 
intervention for those who have decreased performance in daily activities mainly due to 
decreased muscle strength in the shoulders and elbows. 
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Abstract. This article presents the ethical and responsive governance process 
designed by the Hospitalité Saint Thomas de Villeneuve to address aging in place 
services, care and needed support. We will first describe the genesis of a supportive 
community living services organization and its technical translation into Smart 
Digital Health Coordination Platform. Secondly, we will highlight the Living 
Lab methodological approach to support change. Finally, the results section will 
describe change evaluation program engaged and future work perspectives. 


Keywords: Aging in place - nursing homes - living lab - change 


1 Introduction 


By 2030, the increase of healthcare expenditure related to the baby-boomer generation 
will worsen as the European population grows older. It is therefore urgent to prevent 
frailty risks and deliver health and social supportive services to address elderly complex 
care situations at home. 

On the one hand, the proportion of people over 65 years increases rapidly, and 
European society is challenged to prevent decline of functional capacity by addressing 
user’s needs at home with appropriate and optimal health service packages. On the other 
hand, loneliness, geographical isolation and vulnerability, three social determinants, are 
also known to impact mortality, morbidity and health care system use [1, 2]. Health 
and social determinants thus need to be addressed together for better aging in place 
situations. 

Innovative social and medical coordinated care solutions are therefore required to 
prevent complex situations that lead to institutionalization and nursing homes. This is 
the primary goal of the “Reinforced Aging In Place Services” project, EAIPS, namely 
“Dispositifs Renforcés d’Accompagnement a Domicile (DRAD) in France, a French 
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experimentation financed project since 2018 [3]. Due to its organizational major ambi- 
tion, EAIPS-DRAD project is already generalized and integrated into the French code 
of social action and families. It is currently named “Territorial Resources Centre for 
Nursing Homes” (Centre de Ressources Territoriaux Gérontologiques) in the French 
administrative system. Patients benefit, efficiency and medico-economic evaluation are 
three major criteria currently being evaluated in the EAIPS-DRAD project by the DGCS 
(Social Cohesion and Solidarity Public Policies General Direction). 

The aim of this paper focuses on how living lab methodological approaches support 
care activities change in nursing homes. Consequently, this study will try to analyze the 
various acceptance determinants of a smart digital coordination platform designed to 
reinforce aging in place services. 


1.1 From Nursing Homes to Reinforced Aging in Place Services — Dispositifs 
Renforcés d’ACCompagnement à Domicile 


Aging in place is a major public health issue with medical, economic and social impacts. 
In France, social and medico-social housing facilities are facing an unprecedented crisis 
that legitimates a careful consideration of this model. Indeed, industrial processes applied 
to human care, degradation of working conditions [4], numerous adjustments that have 
been made to cope with covid pandemic situations, their effects and organizational 
impacts are so many crisis indicators and urge for a deep and sustainable model change 
[5]. 

A variety of aging in place solutions currently exist, some are called intermedi- 
ate forms of residential provision [6], with varied types of residential, community and 
supportive services [7, 8]. This term defines a new type of housing which is closer to 
the home than to the social and medical/social sector. EAIPS-DRAD system can be 
considered as a bridge between home and institution, especially regarding of two orga- 
nizational characteristics: its organization “outside the walls” [9] and its provision of 
assisted technological services [10]. 

The choice made by HSTV is to provide an “outside the walls solution”, a form of 
nomadic services brought from inside the nursing home to outside elderly accommoda- 
tions [11]. It represents a multidimensional and multi-professional expertise based on 
different actors (gerontologist, nurses, caregivers of the nursing home) who take into 
account the patient’s desire to remain at home. Additionally, Reinforced Aging In Place 
Services/DRAD are completed by sensors and applications that are likely to slow down 
loneliness and/or possible depression [12]. The aim of this system is to anticipate the 
changing needs of elderly people aging in place, within a structured and coordinated 
local community living service (medico-social and health services). 


1.2 Reinforced Aging in Place Services (EAIPS-DRAD) 


As we get older, health and care needs are not likely to stay progressive and continuous. 
Therefore, RAIPS/DRAD experiments an organizational system whose mission is to 
offer an alternative to admission in nursing homes for the elderly with who might have 
functional or cognitive incapacities but would rather stay home. As a matter of a fact, 
one of the HSTV non-profit organization’s nursing homes observed in 2016 that it was 
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no longer possible to fulfill the growing needs of the nearby population: more than 
300 elderly people were on the waiting list. A survey conducted the same year among 
its residents showed that none of them would have chosen the nursing home if they 
could have stayed at home. The idea of a “nursing home outside the walls” service was 
developed and co-designed with staff and families. It has been experimented since 2018 
within 5 institutional home care of HSTV group, and based on three principles: 


1. a complementary approach towards existing community living services: the focus 
is to integrate and strengthen the various coordination levels among professionals 
working for the elderly. 

2. amobile team that responds to the elderly people’s safety needs: gerontologist experts 
in complementarity with other actors such as gerontology care assistant, occupational 
therapist, psychologist; 

3. a 7 days a week and 24 h a day accessible alert system: securing the home through 
the use of smart sensors and the nursing homes professionals in case of emergency. 


These efforts to promote new types of living spaces for older people are not new but 
to date, there was no coordinated and evaluated deployment in France. Today, nearly 
150 people benefit from RAIPS/DRAD services within 5 nursing facilities. 

From now on, because this experimental phase has been positively evaluated, this 
plan can officially provide aging in place preventive support inside the walls of the 
nursing home. These are the new missions called “Territorial Resources Centre for 
Nursing Homes” (Centre de Ressources Territoriaux Gérontologiques). April 27, 2022 
bylaw! gives precise specifications for these missions. 


1.3 Smart Digital Health and Coordination Platform - PALOMA 


Thanks to the non-profit organization executive board and local council of HSTV, Brit- 
tany IT Services and IT Consulting (GCS E-santé) and the Regional Heath Authority 
(Agence Régionale de Santé de Bretagne), the PALOMA smart coordination platform 
was born. Its goal is to enable people with a loss of independence to stay at home with 
the help of the nearest Territorial Resources Center (CRT). Placed at the center of the 
system, the nursing home, is the resource center, in the sense of CRT. 

The challenge of developing such a platform is associated with the tremendous 
changes that health and medico-social nursing homes sector are facing. Their missions 
are progressively evolving from residential settings to Territorial Resources Centers 
(CRT) for the elderly populations living nearby. Their aim is to provide responses to the 
changing needs of frail seniors, primarily by acting preventively and curatively to allow 
them to stay at home by providing support. Two major issues must thus be addressed in 
the smart digital platform conception: 


e Professional coordination: better communication in order to adapt services to the 
various and complex care situations. 
e Ethical management: use of health data in order to predict frailty aggravation. 


! Arrêté du 27 avril 2022 relatif à la mission de centre de ressources territorial pour les personnes 
âgées - Légifrance (legifrance.gouv.fr). 
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The scope of the PALOMA platform project therefore includes the support and the 
deployment of CRT by offering a global, interoperable and predictive digital tool (see 
Fig. 1). 

The digital solution has been designed to help elderly people maintain their inde- 
pendence and stay at home by improving the coordination of social, medico-social and 
health sector players in order to: 


e Provide better support for frail seniors, to preserve social and family networks by 
aggregating a set of existing but dispersed solutions. 

e Reduce hospitalizations through the use of sensors and IoT, telehealth, telecare and 
the use of predictive algorithms and machine learning for trusted and robust decision- 
making. 


Monitoring — alerting - coordination — tele health — frailty assessment & 


o prediction Frailty 
Remote monitoring assessment & 
and alerting w| prediction 


Coordination 
l Tele-health 


Interoperability 


/ Kui | Q 
O.a e Ee o- BB -oote CKO 


Fig. 1. Smart digital health and coordination platform -PALOMA 


Thus, this platform aims at acting on 4 dimensions: 


1. Securing the person’s environment 

2. Managing complex care situations and provide support for caregivers 
3. Enhance coordination and provision of geriatric expertise 

4. Quality of the life project, social isolation and loneliness management 


PALOMA smart digital health coordination platform aims at providing continuous 
support from primary signs of frailty in everyday life, complex care management while 
aging in place (RAIPS/DRAD) and to institutionalized care in nursing homes. 


2 Literature Review 


PALOMA’s platform development is a strategic priority for the Hospitalité Saint-Thomas 
de Villeneuve non-profit organization. The HSTV’s living lab is in charge of coordinat- 
ing the development stages, allocating the necessary resources, undergoing the quality- 
assessment process as well as monitoring the pilot sites deployment. In order to ensure 
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strategic alignment between technical development and supportive community services, 
HSTV’s ethics committee has been consulted throughout the project. The innovative and 
methodological approach is developed hereafter. 


2.1 The Living Lab Approach Implemented at the Hospitalité Saint Thomas de 
Villeneuve 


In 2021, a five-year strategic project on user experience, attractiveness and innova- 
tion has been launched with HSTV local council approval. To implement, develop, and 
structure an innovation culture, a living lab project has been launched the same year. It 
aims at ensuring open innovation policy, attached to the value of the HSTV non-profit 
organization. 

Several definitions of Living-labs co-exist [13]. Forum LLSA2 in France, and 
ENoLL? in Europe both describe living labs as open innovation ecosystems. In HSTV’s 
living lab, the methodological approach follows the core principle of: “open innova- 
tion” carried by the user, “co-creation” and experimentation of the uses of products and 
services, “multiple stakeholders” and the importance of “real-life environment” with 
the aim of defining and developing new products, services, public and community sys- 
tems or new business models [14]. Livings labs thus constitute a new research paradigm 
based on a participatory dimension through collaboration with end users and a mode of 
innovation described as open. 

Considering living labs as potential sources of innovation stimulation [15], a strategic 
priority project of Living lab has been structured within HSTV in 2021. Firstly, support 
from the Forum LLSA has been requested in order to design the value creation and the 
raison of being HSTV’s living lab, as well as elaborate its governance, development 
process and economic model [16]. Secondly, in 2022, a complementary approach has 
been implemented with the training program provided by the Ensembl] Living Lab in 
Lille Metropolitan area. In a dynamic and learning process, co-designed methodology, 
animation postures, evaluation, ethical and governance process of the living lab project 
has been structured. Thirdly, a working and learning expedition during autumn 2022 
with the directory and scientific board of the University of Lille has contributed to rely 
on a definition of living lab activities as “ a process of special attention paid to enabling 
environment in inclusive space” [17]. 


2.2 The Purpose of HSTV’S Living Lab: Supporting Change Activities in Health 
and Medico-Social Institutions 


The structuration of the HSTV Living Lab activities previously described led to define 
its raison of being, which is to meet the needs of change management for HSTV’s care 
and medico-social facilities for change support. A value proposition creation has been 
established to take up to the challenges faced by HSTV’s health and medico-social 
institutions: 


a https://www.forumllsa.org. 
3 https://www.enoll.org/about-us/ 
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1. Bring back meaningful co-activity between the worker and the person that benefits 
from the service to create a dynamic of value creation. 

2. Co-design with stakeholders the useful effects of work change: organizational 
performance, health at work, relationships quality and social cohesion. 

3. Measure effects, results and dynamic processes engaged with change as well as their 
social impact. 

4. Capitalize knowledge and lessons learned 

5. Disseminate the lessons learned internally and externally 


Change in the health and medico-social sector are numerous, and the ambition of 
the living lab is to go beyond a time-limited project management mindset to contribute 
to a “performance of use” [18]. This concept relies on centrality of work which leads to 
the opportunity of value proposition and contributes to achieve performance. 


2.3 How Technological Artifact PALOMA Processes Organizational Change? 


Firstly, to move from a project management logic to the emergence of performance of 
use concept, we propose to challenge our activity model change with simple questions 
[19]: 


e The change process: what is the desired outcome from carrying out this change 
activity? 

e The process of technical and economic transformation: by what means are the subjects 
performing this activity? What kind of performance of use is able to support and 
achieve these goals ? 

e The process of organizational transformation of work: who is responsible for what, 
when carrying out this activity and how are the roles organized? What kind of work 
activity divisions and distribution are in tension and need to be discussed? 

e The process of cultural transformation: what values, cultural norms, rules or 
regulations are governing the performance of use? 


2.4 Ethical and Responsive Evaluation Program in the Design Process of Smart 
Health Digital Services for Users 


Secondly, to support change requires appraising the effects of a complex situation. Based 
on Stake’s “Responsive Evaluation” model [20], the evaluation approach associates 
the experiences of patients and beneficiary and family members for shared decision 
making, and collective participation. Additionally, the evaluation approach focuses on 
the educational, learning and empowerment processes rather than the solely measures 
of effects and outcomes of the change process. 

In the HSTV’s living lab, evaluation process was reduced to four steps, including: 


. Key performance indicators monitoring: i.e. the effects and impacts produced 

. Collective skills acquisition (e.g. autonomy level) 

. Organization capacity to learn, adapt and change 

. Environment and organization capacity to allow people to expand their possibilities 
of action and their degree of control over their task 


BRWN re 


Ambient Assisted living Service Conception in Nursing Homes 161 


2.5 Multilevel Acceptance Framework Assessment for Smart Health Digital 
Services for Users 


Thirdly, to support Ambient Assisted Living (AAL) product and service conception, 
user needs requirements and activity limitations assessment are two crucial components 
to target independent living. AAL system acceptance requires a micro level of sys- 
tem acceptability combined with a macro acceptability level for innovative practices 
acceptance [21]. 

To support change within a complex activity system, an “expansive learning cycle” 
involving several phases is initiated (cf. Fig. 2) and aims, through co-construction, at 
elaborating the bases of new practice decision and collective participation. 


7. Consolidating the new 
practice 


6. Reflecting on 
the process 1. Questioning 


5. Implementing the new 


cies 2. Analysis 


4. Examining the new 3. Modeling the 
model ~~ new solution 


Fig. 2. Sequence of learning steps in a development cycle by expansive learning [22] 


3 Living Lab Approach: A Responsive and Ethical Evaluation 
Program to Co-Create the Paloma Platform 


We built up a responsive ethical program. We proposed to analyze the co-design work 
of the platform through the prism of these different ethical evaluation stages. Indeed, 
the Living Lab participants (users and medical and health professionals) questioned the 
meaning of work activities and service relationship during the design of the smart digital 
platform. In this paper, our aim is to describe change evaluation program engaged in order 
to better apprehend organization change. This reflexive approach, led to the identification 
of a four-stage research-action framework which we propose to describe here: 


1. First, the Ethics Committee group referral (step 1) 

2. Second, controversies raised by the use of the platform (step 2); 

3. Thirdly, digital design conception principles associated with the platform (step 3) 

4. And, finally, use case design to engage discussion on digital platform conception 
principles with the ethical committee (step 4) 
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3.1 Step 1: Ethics Committee Group Referral 


The development of the PALOMA platform raises a certain amount of ethical questions. 
In 2022, two members of the Living lab, a gerontologist and a Nursing Home Director 
of HSTV group, wrote down a document to raise these issues. A synthesis of the referral 
document identified the risks of opposition between HSTV care values and the digital 
platform (Table 1 below). 


Table 1. HSTV care values and risks of opposition summary raised by the digital health and 
coordination platform -PALOMA 


HSTV care values Risks of opposition raised by the 
Platform 
1 Inclusion of users Exclusion through illiteracy? 
2 Home care support Medicalization, surveillance: 
institutionalization? 
3 Aging in place management of complex care Distorted perception of reality? 
situation 
4 Means’ adaptation Blind and globalized 
digitalization? 
Relationship involvement Dehumanization? 
6 Care freedom Care slavery? 


These 6 questions addressed to the HSTV Ethic Committee group aim at providing 
ethical guidance for smart platform specification development. They particularly points 
out strategic dimensions such as: 


Outcomes expected from the use of the platform? 

Values, norms and rules of HSTV cultural group compared to the digital platform 
service production? 

Work division and roles addressed by the digital platform? 

Lessons, knowledge and skills that can be learned and practiced with the digital 
platform? 


3.2 Step 2: Controversies Arising from the Use of the Platform 


In order to materialize these ethical questions associated with the use of the PALOMA 
platform, a collective workshop was organized. Several Living Lab community users 
contributed to co-design the digital platform controversies: HSTV’ representative (health 
professionals, doctors, UX designer, innovation manager, nursing home director, infor- 
mation system director), Aract Bretagne Association (work condition non-profit organi- 
zation), France Asso Santé Association (patient non-profit organization), and Telegrafik 
(smart digital platform software company associated in the Paloma development). Work- 
shop objectives were to collectively debate on digital and Artificial Intelligence health 
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service design principles conception for healthy aging. The controversies were designed 
based on the ethical principles of medical, care and digital Artificial Intelligence (AI) 
project conception design’: 

The workshop was held during the Britany regional conference on Ethics and Arti- 
ficial Intelligence in Health organized by the Ministerial Delegation for eHealth (DNS) 
and Regional Health Authority (Agence Régionale de Santé de Bretagne) on December 
8, 2022. An invitation workshop based on “Digital data use in health to support healthy 
aging” was launched. 

50 people took part in the workshop (12 men and 38 women): 26 caregiver stu- 
dents, 24 health professionals or user representatives. All of them had a previous hour 
conference on AI concepts in Health provided by Ministerial Delegation for eHealth. 

The controversies used during the workshop were: 


“TA is big brother anywhere and anytime at home” 

“TA favors human relationships” 

“IA improves working conditions and quality support for elderly disabled” 
“TA is a matter of experts” 

“IA favors energy sobriety” 

“Aging in place service support will be better if IA collects all health data”. 


3.3 Step 3: Ethical Shared Principles Synthesis During Collective Participation 
on the Debate of Smart Digital Platform Conception 


During the workshop, a facilitator welcomed the participants and invited them over 
a period of one hour, to discuss the controversial issues. As an introduction, a short 
brief on the technical aspects of the platform (predictive algorithms of frailty risks 
signals) was provided. The objective of a collective debate on ethical issues, work impact, 
quality of care with IA use was explained. According to the concept of “moving debate”, 
participants had to occupy the workshop space according to their opinion: the “agree” 
on one side of the room, the “disagree” on the other. They were then invited to express 
out loudly, as many arguments as possible and to switch camps if they agreed with the 
argument expressed. 

Each argument, from one side or the other, was stated successively, trying to rally 
the opposite side. The arguments exchanged were successively noted on post-it notes 
by a co-facilitator on a paper-board: the arguments that led to agreement were noted in 
blue whereas the arguments that led to disagreement in green. 

During the workshop, the ethical design principles, as defined by the DNS in the 
“ethic by design guide”, were explained. A categorization was then realized: the post-it 
notes were grouped in respect of the ethical principles guidelines. Eight ethical principles 
for the digital platform conception were identified: 


1. Enabling discussion: shared decision making and collective participation for the 
platform conception 

2. Consent: informed consent as preliminary use of the platform 

3. Human guarantee: humanized relation and personal link in all relations with the 
platform 


4 Ethic By_Design_Guide_VF (esante.gouv.fr). 
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4. Usability: easy to use and accessible platform 

5. Value: non-commercial purpose and energy sparing culture to be developed 

6. Quality of support: working conditions and care quality support conditions to be 
integrated to address complex care situations, 

7. Respect of privacy: a platform that sets limits to the collection of health data. 


3.4 Step 4: Use Case Design to Engage Discussion with the Ethical Committee 


In order to get a concrete representation of the potential risks of the smart digital platform, 
4 use cases have been designed. They consist of 4 different potential types of data use 
by 4 types of elderly and professional personas (i.e. fictional character). Each of them 
has been designed with the help of healthcare professionals (1 gerontologist, 2 nurse 
coordinators in the RAIPS Project, 1 nurse coordinator in a nursing home). To perform 
the data use case design, several criteria have been described: 


— The persona story and way of life (personal characteristics, assistance plan, past 
medical history) 

— The inclusion in the smart digital platform (context of the persona data use) 

— The organizational activity of the digital platform simulation (what is happening? 
Who is responsible for what, when does the situation occur, and how are the roles 
organized? What is the platform doing and what kind of results does it provides? How 
does the professional in charge take care of the situation?) 

— Worst case scenario: simulation of what the worst could happen? 

— Ethical issues: what are the care values, rules or regulations at stake? 


These smart digital health use-cases were presented at HSTV Ethic Committee group 
on May 24, 2023. They intended to provide ethical discussion, guidelines and specifi- 
cations of Paloma’s platform data use with a user-centric perspective. They also address 
future possible work conditions and quality of care impacts. Finally, these platform data 
use cases highlight the responsibility and governance processes that HSTV non-profit 
organization needs to develop. 


4 Conclusion 


A smart digital platform conception is not limited to technical infrastructure construction. 
As HSTV chose to develop its own organizational settings for aging in place, the nonprofit 
organization is being challenged with the implementation of a living lab approach to 
address ethical issues that digital community service conception raises. 

This papers tried to examine the impact of a technological artefact on the professional 
activity changes associated with the conception of a smart digital and coordination 
platform named PALOMA. Involving professionals stakeholders to decision making 
process and digital co-conception is still a hard work to do because it implies to put 
in the spotlight new work identities, different nature of professional relations, ways of 
managing information, new values and norms elaboration, responsibility and missions. 

Further work will integrate methodological material (description of use-cases and 
personas) and results data (ethics committee specifications; work activity scenario simu- 
lations, governance mechanism identification). Measurement of sustainable work condi- 
tions, quality of life, professional and environmental empowerment will be the our next 
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step to evaluate if those determinants can be achieved within the smart digital health and 
coordination platform. 
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Abstract. The Assistive Technology supply is highly fragmented and almost non- 
existent in many middle and low-resource settings with limited purchasing and no 
service delivery systems. In the Dominican Republic, despite the existence of a 
legal framework on disability, there is no comprehensive source of data about the 
availability of Assistive technology devices. Therefore, the purpose of this study 
was to investigate about the access, use, barriers to access and users’ satisfaction 
of Assistive Technology (AT) in the province of Espaillat, in Dominican Repub- 
lic. The WHO’s rapid Assistive Technology Assessment tool (rATA) survey was 
implemented as a stand-alone technique to 15 outpatient of rehabilitation services. 
Despite the small sample size, the information gathered provides valuable insight 
into the use of assistive products in the Dominican Republic. The findings of 
the study highlighted cost-related factors influencing AT access and low level of 
satisfaction. 


Keyword: Assistive technology - Dominican Republic 


1 Introduction 


In actuality, 1 billion people need assistive technology (AT) to lead productive, inclusive 
and dignified lives, but only 1 in 10 people globally have access to the AT they need. 
Access to AT is essential for many people to maintain and improve function, health 
and well-being, and to participate in education, work and social activities. Among the 
people who commonly need AT are older people, people with disabilities and people 
living with chronic conditions. As the world population ages and the prevalence of 
noncommunicable diseases increases, the need for AT will continue to rise [1, 2]. 

Few statistics have been systematically gathered in countries to show the demand 
for AT among national populations and the need that is currently unmet, despite the 
urgency and the worldwide imperative to enhance access to AT. [3] The AT supply is 
highly fragmented and almost non-existent in many middle and low-resource settings 
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with limited purchasing power and no service delivery systems. Historically, most of the 
governments in low/middle-income countries have not prioritized this sector [4]. 

In low and middle income countries, as most of the Latin American countries, assis- 
tive technologies are not universally accessible for a number of reasons, including high 
costs, limited availability, a lack of governance, inadequate financing, widespread lack 
of knowledge, and a shortage of staff with the requisite training [5]. 

In the Dominican Republic, 12.3% of the overall population lives with a disability, 
according to the 2010 Census on Population and Residence, representing 1 of every 
8 people and constituting a total of 1,160,847. Physical and motor disabilities, visual 
impairments, attention deficit disorder, and auditory impairment are the most prevalent 
disabilities in the nation [6]. 

Despite the existence of a legal framework on disability, as in most of the developing 
countries, in the Dominican Republic there is no comprehensive source of data about 
the availability of Assistive technology devices. This can be explained with the fact that 
some of the main providers or leading suppliers of AT products and services are non-state 
actors, and databases are frequently maintained on an organizational basis with rarely 
external sharing (as is the case with NGOs) [6-8]. 

Therefore, for AT purchasers, such as governments, UN agencies, and civil society 
organizations, to decide whether to invest in the infrastructure of the AT supply chain, 
as well as for AT designers and distributors to identify the populations in need of ATs, 
data on Dominican AT supply and demand is required. 

The purpose of this study is to investigate about the access and use of Assistive 
products(AP) in the province of Espaillat, in Dominican Republic, as well as the bar- 
riers to access and users’ satisfaction with products provided. Furthermore, the study 
aimed to gain more insight into the methods people use to meet their needs for AT and 
produce evidence on the strengths and weaknesses of the current system by choosing 
a target population of community-dwelling individuals attending rehabilitation services 
provided by the local public healthcare system. 


2 Method 


With a target demographic of 15 outpatient of a public rehabilitation center, the WHO’s 
rapid Assistive Technology Assessment tool (rATA) survey [9] was implemented as a 
stand-alone technique. 

The selected rehabilitation center is a public rehabilitation center of Espaillat 
province, and it is in charge of providing rehabilitation services for people with all 
kinds of disabilities, including acquired brain damage, spinal cord injury, neurologi- 
cal, orthopedic, rheumatologic, and post-intensive care unit rehabilitation, among many 
other diseases. According to this viewpoint, its rehabilitation services are responsible 
for managing all functional domains addressed by the rATA (mobility, vision, hearing, 
cognition, communication, and self-care). 

The study received ethical approval from the Yonsei University Institutional Review 
Board. All surveys were conducted through face to face interviews in January-February 
2023. 
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3 Result 


In total, 15 individual responses (73,3% females; 26,7% males) were collected in the 
period January-February 2023. Overall, most respondents lived in rural areas (73.3%) 
and were adults aged 18-64 years (66,7%), followed by adults 65 + years (33.3%). 
(Table 1) A closer examination of the data revealed that the distribution of AT users 
across the Espaillat province geographic area was well balanced, with no discernible 
differences across districts. 


Table 1. Demographic data 


Demographic data N % 
Gender 

Male 11 73.3 
Female 4 26.7 
Age 

2-18 0 0.0 
19-64 10 66.7 
65+ 5 33.3 
Location 

Rural 11 73.3 
Urban 4 26.7 


The higher use of AP was reported for individuals with a functional limitation in 
the mobility domain. The most commonly reported AT product in use was the cane (7), 
followed up by manual wheelchair (3), rollator (2), orthoses for lower limb (2), glasses 
of low vision (2) and axillary elbow crutches (1) (Table 2). 

In terms of the sources to access the AP, they were most commonly obtained from 
a family member or friends (46.7%) or the private sector (26.6%) followed by the non- 
government organization (NGO) sector sources (13.3%), while only 6,7% were obtained 
through the public sector sources or were self-made. The majority of the users (33.3%) 
paid out-of-pocket for their AP(s) or relied on family/friends (46,7%) and only 13.3 and 
6.7% were paid by government funding or a NGO respectively. Most AT users traveled 
less than 5 km (40%) or 6-25 km (33.3%) to obtain their AP(s) (Table 3). 
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Table 2. List of used assistive products. 


Assistive product 


Canes/Sticks, Tripod And Quadripod 


Manual Wheelchairs - Basic Type For Active Users 


Rollators 


Orthoses (Lower Limb) 


Spectacles; Low- Vision, Short/Long Distance/Filters Etc. 


Axillary Elbow Crutches 


Blyn|N vniwlralz 


Note: The responders could select up to 3 products they were currently using 


When asking the users about their satisfaction, the majority (53%) of the users 
reported being dissatisfied with their AP. And when it comes to the unmet need, the 
most commonly reported reason was ‘cannot afford’ (40%), followed by AP unavailable 
(33.3%), lack of time (13.3%), lack of support (6.7%), and AP not suitable (6.7%). 


(Table 3). 


Table 3. Assistive Product Information 


Assistive Product Information N % 
Sources 

Public sector: Government facility, public hospital 1 6.7 
NGO sector: Non-profit facility 2 13.3 
Private sector: private facility/hospital/clinic/shop/store 4 26.6 
Friends/family a 46.7 
Self-made 1 6.7 
Payer 

Government 1 6.7 
NGO 2 13.3 
Paid out-of-pocket (self) 5 33.3 
Family/friends i 46.7 
Distance traveled 

<5 km 6 40.0 
6-25 km 33.3 
26-50 km 1 6.7 


(continued) 
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Table 3. (continued) 


Assistive Product Information N % 
51-100 km 0 0.0 
>100 km 1 6.7 
Do not know 2 13.3 
Satisfaction with AP 

Very dissatisfied 1 6.7 
Dissatisfied 8 53.3 
Neither satisfied/dissatisfied 2 13.3 
Quite satisfied 3 20.0 
Very satisfied 1 6.7 
Barriers to Access 

Cannot afford 6 40.0 
AP unavailable 5 33.3 
AP not suitable 1 6.7 
Lack of time 2 13.3 
Lack of support 1 6.7 


Regarding unmet needs, some users mentioned simply needing a replacement for 
the same product they were using, but other products such as powered wheelchairs, ther- 
apeutic footwear, fall detectors, hearing aids, smart phones/tablets/PDA, GPS locators, 
chairs for shower/bath/toilet and grab-bars/Handrails were mentioned. (Table 4). 


Table 4. List of needed assistive products. (Unmet need) 


Assistive product 


Wheelchairs, manual with postural support 


Wheelchairs, electrically powered 


Manual wheelchairs - basic type for active users 


Axillary/Elbow crutches 


Rollators 


bI | ee ee | ee 


(continued) 
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Table 4. (continued) 


Assistive product 


Therapeutic footwear (diabetic, neuropathic, orthopedic) 


Hearing aids (digital) and batteries 


Smart phones/tablets/PDA 
Global Positioning System (GPS) locators 


N 
2 
Fall detectors 3 
1 
1 
1 
1 


Chairs for shower/bath/toilet 
Grab-bars/Hand rails 2 


Note: The responders selected up to 3 products they need but don’t have or have but need a 
replacement 


4 Discussion 


Despite the small sample size, the information gathered provides valuable insight into 
the use of assistive products in the Dominican Republic. During interviews, it became 
clear that we were raising awareness of the wide range of products and services that fall 
under the umbrella term of ‘assistive technology’ as defined by the WHO, and that a 
large-scale national questionnaire would be of significant benefit in raising population 
awareness in the Dominican Republic. 

The findings of the study highlighted cost-related factors influencing AP access. For 
example, among AP users, aside from those acquired from a family member or friend, 
APs were most commonly obtained from private providers and paid out of pocket, 
and the most frequently reported barrier to AP use was high cost. This suggests a gap in 
public provision of AP in this setting, which is consistent with studies conducted in other 
developing countries [10, 11] and indicates that low/no cost AP provision is still limited, 
despite the presence of some Public institutions and NGOs providing rehabilitation 
services in Espaillat province. This could be due to these organizations’ limited resources 
and capacity to deliver at a bigger scale. 

When discussing unmet needs, the majority of users only had mobility assistive 
products and a minority vision products, however products to assist in hearing, commu- 
nication and personal care were reported as needed, but not currently used, highlighting 
the small range of AP that the respondents in this population had access to. 

Similar to previous research in low and middle income countries, mobility and vision 
related APs are used more frequently and their unmet is more commonly reported than 
APs for other functional domains [12]. These results might be explained by a number 
of variables, such as the accessibility of these services in Espaillat and the population’s 
stronger knowledge and comprehension of mobility and vision issues in comparison 
to other areas, given that these functional challenges are generally more evident and 
well-known. 

The reported unmet demands might indicate the need for additional or alternative 
APs in addition to replacements. This may be due to the high cost, unavailability and 
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the lengthy waits for medical examinations, which push users to utilize products that 
are defective or simply worn off. Information on how to obtain new prescriptions is also 
lacking. In addition, because the majority of the items on the funding-approved product 
lists are mobility aids, many other domains are neglected. 

Regarding the access to AT products, the recommendation section revealed a low 
level of satisfaction. The most common complaint was about the government provision, 
being considered “insufficient” and “providing low levels of user needs and satisfaction”. 
Respondents also stated that, after charity, the only option was to obtain the AP from 
private companies at a very high cost. A number of respondents stated that they had no 
recommendations to make because they were unfamiliar with the system, emphasizing 
the lack of awareness and education about APs and their services. 

This study had some limitations, such as the small sample size. Another limitation 
is that the findings of this study cannot be generalized to other settings in the Dominican 
Republic because the government institutions, private institutions and non-governmental 
organizations (NGOs) that provide rehabilitation services vary by region. As a result, it 
is recommended that future surveys be conducted in other parts of the country. 


5 Conclusion 


Access to AP for disabled people and the elderly should be considered a national priority 
in order to prevent social inequalities and improve the Dominican population’s quality 
of life. Costs, lack of information, and product availability all work together to create 
significant unmet needs, which is a barrier to fair development and the enhancement of 
general well-being. 

The current country’s AP services are not providing the enough coverage. While 
acknowledging and enhancing existing supply lines, new measures should address the 
lack of understanding regarding AP. 

In order to get the right solutions for the provision of assistive technology in 
humanitarian contexts, it is needed to develop products that can be easily sourced, 
environmentally friendly, and easily maintained. 

Based on the current findings, actions should be taken to promote the collection 
of data on AP access in order to determine the current state of AP systems across the 
country and lay the groundwork for system improvement. Additional research is needed 
to determine how to fulfill demand in the Dominican Republic in order to solve the huge 
gap in access to appropriate assistive technology. 
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Abstract. With the new Industry 5.0 future factories can effectively face the aged 
workforce challenge, making workflows more enriched and flexible and capable 
to increase work well-being. This paper described how self-quantified worker 
could be a successful tool to achieve with this with a careful collaborative design. 
Our vision aims at empowering the aged workers and engage them with the work 
community based on adapting the factory shop floor routines to their changing 
needs while they age and support the aged worker to understand and develop 
his/her own competence. 


Keywords: Self-regulatory strategies - work wellbeing - aged workforce 


1 Introduction 


The European workforce is rapidly ageing. In 2030, workers aged 55—64 will make up 
30% or more of the workforce in many countries, while the phenomenon of workforce 
shrinkage phenomenon is also evident with a decline of 0.4% every year between now 
and 2040 [1]. As a consequence, many governments are planning to raise the official 
retirement age, resulting in longer working lives with direct effects on the management 
of the workforce of companies. There is evidence that sustainable long-term employment 
can be associated with positive health outcomes if working conditions are appropriate 
[2, 3]. This is especially relevant in the case of industrial workplaces, as they appear to be 
increasingly dependent on the skills, knowledge, and experience of mature workers, and 
they must maintain the workability and employability of workers with adaptable working 
conditions. Industry 5.0 is booming to put workers at the core of the technological 
revolution initiated a decade ago by Industry 4.0, facilitating worker positioning to more 
complex activities such as abstraction and problem solving. Although Industry 4.0 could 
make an intelligent decision through real time communication and cooperation between 
manufacturing things thanks to Artificial Intelligence (AI) and the Internet of Things 
(IoT), Industry 5.0 introduces the principles of social fairness and sustainability in the 
digitalization and AJ-driven technologies process initiated a decade ago, to highlights 
the importance of human and machines cooperation [4]. 

This new paradigm enables workers, and specially aged workers explore new oppor- 
tunities for expanding their capabilities by the qualitative enrichment of their work, 
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changing their role to supervision and solving unexpected problems, a position where 
the aged workers can find major benefits from the experiences and skills. However, 
fast-evolving labor conditions are also a source of daunting challenges for mature work- 
ers. They need from a friendly, flexible work arrangement in the work environment to 
help mature and experimented workers cope with their intrinsic and extrinsic body and 
life changes as years ago by and while working places introduce new workflows and 
technologies. 

The combination of the digitalization process initiated during Industry 4.0, the vision 
of the human center of Industry 5.0, and the adaptation of industrial environments to 
occupational safety and health (OSH) practices enable the promotion of adaptable and 
sustainable working lives. Workability enhancing models suggest different actions and 
strategies to promote it at different levels [5, 6]. In the case of older workers, most of 
the proposed solutions recommend a stage model that addresses different dimensions of 
daily living of individuals that cover aspects such as health, competence and skills, and 
motivation, in the promotion of health at the core. In all of them, health promotion plays 
an essential role, but understanding the work adjustments needed to compensate for 
ageing-related functional changes. Redesigning work tasks according to the strengths, 
needs, and abilities of each individual is crucial to ensure the well-being, productivity, 
and workability of ageing workers, and an important success factor in successful age- 
ing. Therefore, innovative models to promote better health and life expectancy in older 
workers are necessary to improve their opportunities to enhance their later life. 

This paper analyses the collaborative design of a self-quantified dashboard to intro- 
duce improvements in worker well-being and self-track for task performance improve- 
ment and increase efficiency and productivity. This dashboard aims to lead to positive 
behavioural changes to maintain well-being at work and in private life, towards a suc- 
cessful ageing process, allowing workers to play an assisting role in workability and the 
successful transition to an active and healthy retirement. 


2 The Design of the Self-quantified Dashboard 


When designing human-centered solutions in an industrial environment, the individual 
worker point of view is central in the design and evaluation activities. New solutions 
should be designed to meet the real needs of workers while being meaningful and engag- 
ing with their daily work activities. Consequently, the medium- and long-term results of 
the proposed solution should have a positive impact on worker well-being, productivity, 
and workability. 

Our aim is to design a dashboard that supports self-management of mental and phys- 
ical wellness outcomes while aged workers are embedded in an adaptable intelligent 
working environment that supports them with different interventions that support the 
beneficial, productive, and safe ageing process. This dashboard aims to lead to positive 
behavioral changes to maintain well-being at work and in private life, towards a success- 
ful ageing process. The dashboard design aims to adapt the principles of the Quantified 
Self movement to the real needs of the aged industry workers to favour the engagement 
with the adaptation intervention proposed by their managers. Quantified self-movement 
refers to the practices of tracking and self-measurement of personal health data as a 
practise orientated toward self-improvement and healthy lifestyle routines [7]. 
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The initial design involved aged workers from the beginning of the process. Focus 
groups were held with the participation of 16 employees from four different indus- 
trial settings in Spain and Germany, all over 50 years old. During the focus group the 
researchers participating in the session used a funnel approach (REF) to discover the 
workers daily routines, their needs regarding the ageing process and the possibility to 
include ICT to support them or to mitigate negative aspects of the work due to the age- 
ing process. During this first process the unmet needs and requirements about ageing, 
workability and private life-work balance emerge. 


chronisation 


Fig. 1. Main Barriers, Needs and requirements identified during the Focus Group Session 


Based on this initial identification of requirements, a hierarchical representation of 
goals, questions, and measurements discovered during the session with the workers were 
described using a GQM model [8], and based on this model, designers proposed a first 
visualization mockup of the dashboard. This first design included three aspects con- 
sidered for a successful ageing process: physical, cognitive, and social/mental domains 
that were in alignment with the needs and requirements discovered during focus group 
sessions and organized in the GQM model. In this sense, the three domains were pre- 
sented separately for the work environment and for private life. Following the approach 
proposed by Hall, the dashboards build on common and well-established self-tracking 
approaches. Based on the data automatically collected by worker personal devices 
(smartwatch and smartphone), feedback and environmental context awareness perform 
automatic detection of the worker’s behavior pattern and stimulates it to self- challenges 
and encourages him to achieve the goals. 

Once the first mockup was designed, a set of collaborative workshops were per- 
formed. In these workshops, a variable number of age workers, designers, and technicians 
participated to iterative co-create the final version of the dashboard. In the workshops, 
the participants used the available mockup to highlight the important data, prioritize and 
link them, using a pre-attentive processing technique. Thanks to this, the participant was 
able to iteratively identify the graphical properties of each piece of information; hence, 
the mockup was refined and improved between workshops. In addition to the graphi- 
cal representation, the participants discussed and selected self-quantified measures that 
better improve workers’ performance towards goal attainment. 
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During the workshop sessions, field notes were collected, while suggested changes 
were made to the prototype to continue discussion. After each of the workshop sessions, 
the notes were reviewed looking for common preferences among the participants, which 
were incorporated into the dashboard design, and it was used as a starting point for the 
next workshop session. 

The emotional status of the worker plays a key role in the dashboard due to its 
influence in workability and private life. To avoid lack of participation or negative atti- 
tudes toward quantification of their roles due to the influence of emotional status, self- 
regulation techniques based on goal attributes were proposed to support self-qualified 
approaches. In this way, the individual worker proposed his/her own goals, allowing 
workers who set specific challenges to appreciate the qualified self-characteristics. This 
simple technical aspect has the potential to make workers perceive their change more 
positively and take greater benefits from it [9]. 

To implement this approach, a dedicated algorithm was designed and employed. This 
algorithm automatically monitors the level of achievement of each goal using the data 
gathered by smartphones and smartwatches mainly, but other sensors network available 
on the factory shopfloor could be also added, and according to the gathered data and the 
moment of the day, different motivational messages are sent to the users to encourage 
them to work harder or maintain this behaviour. Ideally, workers should achieve their 
challenges not only on the physical level, but also on the socialisation level, so moti- 
vational messages are aimed at balancing these life aspects. The rules on the delivery 
of motivational messages were discussed in the different co-creation sessions, with the 
aim of guaranteeing a fair and stimulating motivational system. In these, particularly, 
motivational messages always try to show positive aptitude. 

The final version of the worker dashboard consists of 4 main sections presented 
below. 


2.1 My Messages 


This section aims to present notifications and messages to the user. This directly addresses 
the need for recall and awareness identified during the focus groups. Even this section 
is not really a graphical dashboard functionality, workers demanding the need of having 
a global view of those aspects that should be considered during the day, as well as 
reminders that contribute to the long-term positive habits acquisition. The section has 
the basic functionality of a mailing box, so they can manage the messages with the 
following features read, filter, and delete. Different types of messages are shown using 
different types of icons that have been chosen after research on the different types of 
information to be shown. 


2.2 My Daily Habits and My Job 


My daily habits and My job are the core sections of the dashboard, and they are intercon- 
nected, hence, workers can perceive the influence of their daily habits on their workability 
and on their private life. The emotional state of the worker plays a central role in this 
evaluation and follow-up. The graphical representation of the data shows a relative scale 
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that aims to show the level of achievement of the personal goals. The graphic repre- 
sentation is supported by a colour scale that represents a low, medium, or high level of 
achievement, which also depends on the time of day. In this way, the level of achieve- 
ment and motivational messages are adapted to the daily routine of the user. Additional 
textual information is provided to support the interpretation of the data and contribute 
to employee motivation and long-term positive habit acquisition. 

In support of the self-quantified strategies adopted in the dashboard, a self-assessment 
questionnaire has been included in the My Job section. This questionnaire attempts to 
collect additional information from the user, in terms of stress, concentration, tiring after 
work, and pain during the workday, and allows dashboard algorithms and early detection 
of worker problems that can affect your workability or private life. The information of 
these variables is also presented in the form of a trend graph, supported by a three-colour 
scale (from red to green). 

Finally, the dashboard presents a set of graphical information that supports workers 
with ergonomic assessment of the job position, hence preventing physical problems and 
work breaks, and prevent the detection of situations that cause stress, fatigue, and lack 
of concentration. 


2.3 My Goals 


The My Goals section is the core of the self-regulation theory implemented in the dash- 
board. Thanks to this section, workers can personalise their goals and better manage 
their daily habits. Additionally, goals are used to personalise the visual analytics and 
feedback messages that workers receive during the day according to their achievement 
of their goals (Fig. 2). 


My daily habits 


Fig. 2. Different versions of the My daily habits sections developed during the collaborative 
workshops 


3 Evaluation Results 


User acceptance was measured using a self-developed acceptance questionnaire with 
the participation of real potential users. The questionnaire consisted of 11 items on 
the following dimensions: Perception of ease of use, perceived usefulness, intention of 
behavioral use, and self-efficacy. Items are rated using a 5-Likert scale (0, Extremely 
disagree-extremely agree). In addition, acommon-based UTAUT questionnaire was also 
used. A total of 125 workers over 50 years participated in the evaluation. 
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Almost all participants perceived the system as useful for their daily tasks at work. 
The functionalities proposed in the dashboard were considered useful and covered the 
needs and barriers identified by the workers. Participants indicated their desire to use 
the system and they considered that they would not have significant learning problems 
using it properly. 

The results (Table 1) also clarified some subjective statements. Feedback on the 
following items was particularly positive: (1) The tool was clear and easy to understand, 
(2) receiving recommendations through the tool would provide you useful measurements 
of your health (for example, your cardiac rhythm), and (3) you can use the tool if someone 
shows you how to do it first. 


Table 1. Results of the acceptance questionnaire 


Dimension Statement Mean (SD) 
Usability The tool was clear and easy to understand 3,7 (0,44) 
Usability It would be easy for you to interact with the tool 3,5 (0,49) 
Usability It would be easy for you to learn to use the tool 3,55 (0,49) 
Acceptance Receiving recommendations for the tools would provide you | 3,57 (0,54) 


useful measurements of your health (for instance your 
cardiac rhythm) 


Acceptance Using tool services, such as ergonomic assessment, would 3,53 (0,61) 
help you address effective emergency situations in your work 


Acceptance Advice/recommendations during the working day would be | 3,32 (0,71) 
useful as a motivation to change your health habits whenever 
is necessary 


Behavior intention | If possible, you will try to use such the proposed solution 3,31 (0,71) 


Self-efficacy You can use the tool if someone shows you how to do it first | 3,36 (0,75) 


The UTAUT results (Fig. 1) show that the more positive domains were performance 
expectancy (the degree to which an individual believes that using a system will help 
achieve gains in job performance) and attitude (the person’s acceptance process) while 
the less positive domains were social influence (the degree to which others believes will 
affect someone to use the new system) and willing to pay (Confidence, commitment, 
and motivation to accomplish to acquire the solution) (Fig. 3). 
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Fig. 3. UTAUT questionnaire results 


4 Discussion 


More than 200 participants, designers, developers, experts in labour health, and final 
users have participated in the co-creation process presented in this paper. As a result 
of this work, a dedicated dashboard has been successfully implemented to support and 
accompany aged workers in their ageing process, so that they can maintain their abilities 
and develop their potential. The results of the needs and requirement elicitation study 
show that older workers have positive intentions to use technological support that mit- 
igates the impact of hard work conditions on their physical and emotional health and 
are willing to stay active as much as possible if they have good health conditions. The 
workers appreciated the increased sense of early detection of potential work issues that 
negatively affect both their workability and their health. 

Close cooperation with aged workers gave us valuable insight into the discovery of 
the critical requirements for both the development and implementation of the dashboard, 
but also facilitates the development of current demographic change policies. 

Workers indicated their willingness to use the system since the results improved the 
well-being even though there is little evidence of the technological development of older 
workers; this agrees with the finding from [10-12]. Furthermore, in this study, most of 
the 125 workers reported very good acceptance and high satisfaction in terms of usability. 
The influential role of improving the balance of private work on motivation, perception, 
and engagement with the proposed solutions emerged in worker interviews. In this sense, 
critical requirements are related to the adaptation of the working environment to normal 
functional decline. Early detection of physical and mental problems has the potential to 
alleviate factors that negatively impact the workability of these workers, contributing to 
increased job participation and successful retirement. In this sense, this work contributes 
directly to the research agenda proposed by [13], providing older workers with new 
assistive technology and new age-friendly workspaces. 

One limitation of this study is that, even though the large number of real users 
involved during the whole process, especially at the last stage, during the usability 
assessment, it is a need to corroborate the good acceptance results with acceptability 
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studies using the real setting pilot. The lack of this fully piloted implementation during 
this study allows only a purely descriptive evaluation of the tendencies and potential 
results. Therefore, the high number of real users during the whole process is sufficient 
to refine the dashboard design as an optimal process. Design procedures, made as user- 
centred design as possible, reduce failure risk due to acceptability or usability issues 
during a real-setting pilot. Aspects such as the impact on workability and well-being 
were not addressed in this first study. For this reason, a full pilot of the solution is 
planned, with the participation of more than 100 workers, to show the efficacy of the 
proposed dashboard in supporting the workability and well-being of the affected workers. 

This study is the first milestone on the way to achieve successful intervention and 
technology support systems that have a positive influence on the successful ageing 
process of older workers before their retirement. 


5 Conclusions 


Due to demographic change, there is a need for greater participation of older adults 
in society. Policy developments in industrial countries incorporate work as a means to 
achieve this, but the practical implementation of these policies is required to be with the 
aim of improving the understanding of the aged works, daily routines, and accompany 
them in the ageing process, so they can maintain their abilities, developing their potential; 
this paper proposes the co-creation of a dashboard. 

Drawing upon self-tracking devices, dashboards, and corporative dashboards used 
to monitor workability and productivity, this article uses conceptual and empirical tech- 
niques to guide the research and design work of technological tools supporting the 
workability and well-being of aged workers. The main contribution of this paper is the 
application of co-creation techniques to a more general understanding of the specific 
need of aged workers in workability and well-being supported tools. 
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Abstract. The purpose of this study was to investigate the effects of the Korean 
Smart Home Modification Program (KSHMP) on the activities of daily living and 
health-related quality of life of people with physical disabilities. The study used 
a single-group pre-post design with 10 people with physical disabilities living 
at home. During the intervention period, the KSHMP was implemented, which 
included occupational profile, smart home installation, setup, training, task-based 
feedback, and monitoring. Post-intervention changes in activities of daily living 
and health-related quality of life were assessed with the Canadian Occupational 
Performance Measure (COPM) and EuroQual-5 Dimensions (EQ-5D). After the 
KSHMP, all 10 subjects improved their activities of daily living and quality of life. 
In addition, the occupational performance of all subjects was maintained. These 
results show that a customized smart home has a positive impact on improving 
the activities of daily living and quality of life of people with retardation and is an 
efficient alternative. 


Keywords: Smart Home - Home Modification - Quality of Life - Disabled 
Persons - Korea 


1 Introduction 


People with physical disabilities (PwPD) frequently have difficulty managing and access- 
ing their homes, depend on a caregiver for assistance in completing their daily-life 
activities [1]. Based on the Institute of Medicine model published by Pope and Brandt, 
the degree of disability was reported to vary depending on the individual’s ability and 
support for the physical-social environment [2]. The per-son-environment-occupation- 
performance model is a measurement of the degree of occupational performance, and it 
changes with the appropriate interaction of the person, environment, and occupation [3]. 
It is important to note that an increase in occupational performance can create meaningful 
occupations, promote occupation participation, and improve quality of life [4, 5]. Their 
study suggested that PwPD, with increased occupation-al performance, were associated 
with creating and satisfying independent activities of daily livings (ADL) that could 
further improve their quality of life [6]. Home modification programs include improved 
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housing structure, relocation of furniture, installation of amenities, efficient moving 
space, and education for changing environmental adaptation, and Advanced technolo- 
gies such as smart homes, enable PwPD to improve their ADL and participate more 
in life [7, 8]. The medical rehabilitation system in Korea consists of multidisciplinary 
teams, including doctors, occupational therapists, and social workers, in rehabilitation 
medical institutions, and is implementing a home modification program as part of the 
discharge plan [9]. However, it is difficult to reflect sensor lighting, access outlets, con- 
trol brightness, and use auto doors, or people with stroke and spinal cord injury; ADL 
independency limitations remain in homes [10, 11]. In addition, the requirement for a 
home environment that meets the physical functions and needs of PwPD is increasing 
[12, 13]. A smart home is defined as a technology and service environment that combines 
in-formation and communication technology with human space and devices to provide 
value, such as safety, convenience, economy, and pleasure [14]. Based on research on 
people using smart homes, controllers, such as voice recognition, tablets, and remote 
controls, are key technologies for improving the quality of life of PwPD and can remotely 
control home appliances, phone use, alarm settings, lighting on/off, windows, doors, and 
applications for daily activities. However, this has mainly been conducted abroad [15, 
16]. In Korea, owing to different barriers that limit the use and effectiveness of smart 
homes, such as complex use, high cost, lack of institutional maintenance, and opinions 
deeming smart homes unnecessary, the only available studies that apply smart homes to 
a testbed; clinical studies that apply smart homes to actual homes are limited [9, 17]. 
The objective of this study is to gather information on the applicability, effectiveness, 
and limitations of providing a Korean Smart Home Modification Program (KSHMP) 
to community practice areas. This is the first step in developing programs that promote 
quality of life, occupational performance, and home management. 


2 Method 


2.1 Participants 


From December 2022 to May 2023, we visited ten participants at home and provided 
KSHMP. The characteristics of the participants were 5 men and 5 women, with an average 
age of 46.7 years, five spinal cord injuries, two strokes, and three cases of cerebral palsy. 


2.2 Korean Smart Home Modification Program (KSHMP) 


The KSHMP consisted of three phases: occupational profile, intervention and monitor- 
ing, and follow-up. In the first phase, researchers visited participants’ homes to collect 
baseline information, identify key needs and smart devices in use, assess the occupation 
space, observe, and analyze occupational performance, and explain and discuss home 
modifications. In the second phase, researchers installed and set up the smart home in 
each participant’s home. Researchers trained participants and caregivers on how to use 
the smart home and provided task-oriented feedback and monitoring. In the third phase, 
we conducted a post-test and interviewed participants about their experiences with the 
smart home. The KSHMP was determined by space through a literature review of the 
impact of smart homes on daily life [18]. The applications and contents of the smart 
home in this study are shown in Table 1. 
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Table 1. Smart home by space 


Space Contents 


Living room and bedroom | Smart home appliances [TV, air conditioner, refrigerator, etc.], 
Smart plug, Smart light switch, Get news and weather alerts with 
voice speaker, Listen to radio with voice speaker, Smart window 
opener, Electric lift for wardrobe, Smart curtains/blinds, Smart 
remote control hub, Motion sensor, Temperature and humidity 
sensor, Turn on sleep background music, Smart mood light, Smart 
heating controller, Smart air purifier, Smart humidifier, AI motion 
tracking camera 


Kitchen Gas Detection Sensors, Smart Gas Blocker 

Entryway Smart Door Opener, Smart Doorbell, Smart Door Lock 

Other Smart Speaker, Smart Medicine Box, Pet Robot, Home Smart 
Farm, turn on/off all lights and appliances, Notify parents of 
emergencies 


2.3 Occupational Profile 


The occupational profile is summary of a participant’s occupational history and expe- 
riences, patterns of daily living, interests, values, needs, and smart home relevant con- 
texts. Using a client-centered approach, the occupational therapist collected information 
to understand what participants want and need through SHT and to identify past expe- 
riences and interests that can help them understand current problems and problems. 
The researcher evaluated the participants’ occupational performance, satisfaction, and 
health-related quality of life before and after KSHMP. Canadian Occupational Perfor- 
mance Measure (COPM) was used for occupational performance and satisfaction, and 
EuroQol-5 dimension (EQ-5D) was used for health-related quality of life [19, 20]. 


2.4 Home Modification 


Home modification was performed to improve the participants’ indoor ADL. After 
checking the participants’ bedroom environment, the researcher installed controllers 
and smart devices based on the participants’ physical functions to lower environmental 
barriers and increase convenience [21]. The researcher monitored and supplemented this 
to adapt to the changed smart home environment and improve ADL during the study 
period. 


2.5 Smart Home Education 


The researchers taught participants and guardians to understand and use smart homes. 
It supplemented the manual provided by smart device manufacturers and provided it to 
individuals in an easy-to-understand manner. 
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2.6 Task-Oriented and Simplify Occupational Performance Skills 


In this study, the task-oriented approach is a training method focused on special func- 
tional tasks that combine musculoskeletal for nervous systems, providing occupation- 
based routines for participants rather than repeatedly practicing normal patterns [22]. 
The researcher prioritized meaningful activities for each topic to build a task based on the 
smart home, which was changed for each participant based on the occupational profile. 
However, if the participants first performed a task or failed to perform it independently, 
they could learn how to perform the task accurately by referring to the error-free four- 
step learning in Thivierge et al. [23]. We modified the participants’ fewer effective skills 
to make them efficient and allow them to perform their occupations with less physical 
effort [24]. 


2.7 Statistical Analysis 


Statistical analyses in this study were performed using JAMOVI version 2.3.26.0 soft- 
ware. Participants’ general characteristics and pre-post assessments were presented as 
means and standard deviations. Due to the small sample size (n = 10), a non-parametric 
test, the Wilcoxon signed-rank test, was used to test whether there was a central differ- 
ence between the pre- and post-tests between the two groups. The level of significance 
was set at P < 0.05. 


3 Results 


3.1 Participants and Intervention 


Study participants were recruited based on selection criteria through the Gangwon-do 
Regional Health & Medical Center for People with Disabilities and the Seoul Northern 
Regional Health & Medical Center for People with Disabilities. A total of 10 people 
participated in the study, and their pre- and post-assessment results are shown in Table 2. 
All participants were treated at the Department of Family Medicine and Rehabilitation, 
and the intervention began after they agreed to participate in the study. This study was 
conducted with the approval of Yonsei University Wonju Institutional Review Board 
(1041849-202211-SB-218-02). After receiving training on the application of the smart 
home intervention, a licensed occupational therapist visited the participants’ homes from 
December 2022 to May 2023 to implement the smart home modification program. 
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Table 2. COPM, EQ-5D change Scores for Participants Who Had a KSHMP. 


Evaluation M (SD) 
Initial Assessment (n = | Reassessment (n = 10) | Change (n = 10) 
10) 
COPM | Performance | 2.7(0.8) 7.5(1.6) 5.0 
| Satisfaction | 3.2(0.5) 8.11.5) 49 
EQ-5D Quality of life | 0.47(0.07) | 0.78(0.03) 03 


4 Discussion 


A smart home is a new home environment for people with physical disabilities in the New 
Normal era and is important for health-related quality of life [12]. However, rehabilitation 
experts feel that the smart home is cumbersome and has limitations in its application 
compared to the development of fourth industrial technology. The fast-developing smart 
home is a promising new model for the satisfactory daily lives of physically handicapped 
people. PwPD can experience daily comfort when using a smart home, which coincides 
with a study indicating that an environment suitable for physical ability tends to lower the 
dependence on daily life and increase the health-related quality of life [25]. Participants 
in this study applied the smart home technology to remain in their home environment for 
as long as possible. The occupational therapist evaluated the occupational performance 
in the bedroom based on the occupational profile. Through the occupational profile 
and occupational performance evaluation, uncomfortable activities due to the physical 
limitations of the study participants were observed, and the strengths and barriers of the 
environment were identified and modified. These points were similar to the existing home 
modification procedures [26]. Based on previous studies, home modification has been 
reported to reduce parental burden and housing support costs through the improvement 
of housing structure, relocation of households, installation of convenience facilities, 
secured movement, application of auxiliary devices, changed environmental adaptation, 
and housing applications [27]. In this study, education and tasks related to the smart 
home were provided to allow the research participants to adapt to the smart home; 
occupational performance skills were simplified. All study participants experienced the 
difficulty of learning how to use a smart house; however, they noted that it was essential 
as they became accustomed to it. As can be observed from previous studies, it was 
initially evaluated as cumbersome because of the limitations of smart home applications 
for physically disabled people. However, they became accustomed to the smart home 
and could participate in limited activities despite uncomfortable hands and walking [28, 
29]. Owing to these changes, it was confirmed that they participated in the occupation by 
controlling the lights, stands, fans, heaters, air conditioners, and curtains in the bedroom, 
even with their physical restrictions. 
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Conclusion 


The study identified interventions and home modifications that can assist PwPD in 
managing their condition within their homes. Implementing the KSHMP has the potential 
to encourage the development of cost-effective service delivery models. Utilizing the 
KSHMP to improve occupational performance and health-related quality of life for 
PwPD may broaden the scope of research and interventions used in practice, eventually 
extending to the community for occupational therapists. 
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Abstract. This work presents a real-time system for tracking multi- 
ple object in the context of meal preparation when using the Cognitive 
Orthosis for CoOKing (COOK). This system is called SafeCOOK. It aims 
to provide more capabilities to detect some dangerous situations that the 
current system does not consider. For example, it can locate a utensil 
or other kitchen object that has been left on the cooking surface of the 
stove while a meal is being prepared. This system uses a hybrid method 
based on YOLO and KCF to detect, track and drop cooking utensils 
as they enter and leave the cooking area, and is capable of monitoring 
an entire cooktop in real-time with a single camera. The software has 
been implemented on an embedded platform in the smart stove and has 
been added to it. The system produces good segmentation and tracking 
results at a frame rate of 1 to 4 frames per second, as demonstrated in 
extensive experiments using video sequences under different conditions. 


Keywords: Ambient Intelligence - Object tracking - Multi-object 
tracking - Context-Aware - YOLO - KFC 


1 Introduction 


The availability and accessibility of home monitoring technologies have enabled 
friends, family members, and caregivers to track the activities of older adults [10]. 
Tracking the activities of daily living (ADLs) of older adults is also important for 
measuring the safety conditions of older adults [14]. Smart homes are equipped 
with Internet of Things (IoT) devices that monitor the elderly and communicate 
real-time data/information to stakeholders [2,13]. Ambient intelligence appli- 
cations, such as motion and position detection, fall detection, telemonitoring, 
cognitive technology assistants, sensor-based wearable systems, and automated 
home monitoring, are among the tools to provide safety precautions for older 
adults during their daily activities [9, 10]. 

Among these ADLs, meal preparation is one of the most important activities 
of daily living [6,22]. Especially when used as part of the home maintenance 
assessment for people with or without loss of independence. Activities such as 
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meal preparation have recently been explored to exploit the assistive capabilities 
of ambient assistance solutions to prevent dangerous situations [4,18,22]. In 
the context of meal preparation, we have identified 3 categories of so-called 
dangerous situations. The category of omissions concerns objects appearing on 
the cooking surface, the oven door open, and the burners left on without a pot 
on it [7,14]. In the category of an absence of supervision, it is a question of 
being present in front of the stove during the cooking process. This category 
varies greatly depending on the temperature of the burners, the contents of the 
pans, the temperature of the oven, and the cooking time. The last category is the 
presence in the home during the use of the stove. We will focus on the dangerous 
situations of the forgetfulness category in the following. 

Contextual awareness is particularly important in a high-risk activity such as 
meal preparation, where assistive technologies must be able to identify and assess 
problems as they arise and manage dangerous situations in real-time. Except that 
some elements of context are not observable by motion or contact sensors [3]. 
The integration of computer vision and deep regression network-based systems 
into smart stoves would provide more context to recognize certain dangerous 
situations during meal preparation. The combination of using a different camera, 
IoT sensors, and real-time processing of this data allows for the introduction of 
multiple objects tracking techniques [5, 12,19]. 

Multiple object tracking (MOT) is a research topic related to computer 
vision [11,16,17,21]. When we talk about MOT, it is largely about locating 
multiple objects, maintaining their identity, and generating their trajectories 
from an input video. As a mid-level activity in computer vision, multiple object 
tracking underlies high-level tasks such as pose estimation, action recognition, 
and behavior analysis [11]. It has many practical applications, such as visual 
surveillance, human-computer interaction, and virtual reality [11]. 

In this paper, MOT is used for the detection of objects appearing on the 
top of the stove to improve safety during meal preparation when a round is 
left hot with a cloth on it; (2) the ability to maintain or abandon the tracking 
of detected objects to take into account the evolution of the context. Since a 
dangerous situation can be caused by different types of objects appearing on the 
stove, we developed a real-time algorithm to extract multiple perspective views 
from a single capture device to track objects in real-time and stop tracking once 
the object leaves the camera’s field of view over the stove. 

The remainder of this paper is organized as follows. Section2 briefly 
introduces the COOK system. Section3 presents our object-tracking pipeline. 
Section 4 presents how this algorithm is applied to the COOK system. Section 5 
presents the results and a brief discussion. Finally, Sect.6 concludes the work. 


2 COOK: Cognitive Orthosis for CoOKing 


The Cognitive Orthosis for coOOKing (COOK) is an innovative, context-aware, 
stove-connected smart tablet application designed to optimize the independence 
of individuals with cognitive deficits during meal preparation [7,18]. This appli- 
cation was originally designed to specifically target the cognitive deficits of people 
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with moderate or severe traumatic brain injury [6,14]. But now the system is 
designed for anyone who wants a safe environment for meal preparation. The 
COOK system addresses non-emergency situations. The system assists the per- 
son with tasks related to meal preparation (ingredients to prepare, planning, 
safety rules, etc.). It can detect pre-defined hazardous situations through vari- 
ous sensors, including motion and door sensors, ultrasonic range finders, door 
openings, flame and high-temperature sensors, and smart switches as shown in 
Fig. 1. 


Fig. 1. Example of some sensors installed on the smart stove. 


The main components are (1) a Self-monitoring Security System, (2) a Cog- 
nitive Assistance Appliance, and (3) a Configuration System. Only the self- 
monitoring Security System will be discussed in the rest of this paper. 


2.1 Self-monitoring Security System 


In autonomous operation, i.e. not depending on the other components of the 
system, the Self-monitoring Security System (SSS) has the role of monitoring all 
the so-called dangerous or hazardous situations [1]. While the user is cooking, 
if necessary, the SSS will progressively indicate the presence of a potentially 
hazardous situation that must be resolved within a predefined time frame. The 
SSS is designed to monitor three hazardous situations: supervision, omissions, 
and presence. The safety system collects information from the sensors, detects 
critical errors and hazardous situations, and, if necessary, turns off the stove and 
calls for help. Through the sensor infrastructure, the safety system can detect 
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the temperature of the top of the stove, the presence of a person near the stove, 
the opening of the oven door, the temperature of each burner, the amount of 
electric current flowing on each burner, etc. 

Several standard and predefined security rules are provided by the safety 
system. However, to monitor dangerous situations, the SSS uses sensors such 
as motion and door sensors, ultrasonic distance sensors, flame, and high- 
temperature sensors. These sensors are not able to distinguish between different 
objects that arrive at the burner surface. Some of these objects are kitchen 
towels, hands, cooking utensils, etc. Therefore, the SSS needs a preventive assis- 
tance model that captures each user’s idiosyncrasies, personalizes preventive 
assistance, and provides context-aware assistance regarding food preparation 
hazards. The objective of this version of the SSS is to extend its potential to 
detect new dangerous situations. By adding a camera to the existing sensors and 
a new real-time object-tracking model to the SSS. The following situations could 
be easily detected in conjunction with user context information. 


— Leave the active hotplate empty for several seconds. 

— Forgetting something on the cooking surface. 

— Real-time detection of a hand reaching for a hot surface. 

— Follow in real-time the objects that appear on the cooktop. 


To sum up, to have a module that makes it possible to follow in real-time 
several objects that appear on the cooktop during the preparation of a meal. 


3 Multi-object Tracking 


Predicting the object’s location (bounding box) with class is called object detec- 
tion [21]. By locating the potential object locations in each frame, object detec- 
tion can provide observations for detection-based object tracking. Therefore, 
either in every frame or when the object first appears in the video, each tracking 
method requires an object detection mechanism [11]. 

Object tracking is one such application of computer vision where an object 
is detected in a video. Otherwise, it is the process of identifying the same object 
and keeping track of its location with the unique label as it moves around in 
a video. For example, you have a video of someone cooking a meal, and you 
want to track the location of the frying pan constantly throughout the video in 
real-time by estimating its trajectory. Object tracking can be done using one of 
the two approaches that exist in the field. One is called Single Object Tracking 
(SOT) and the other is called Multiple Object Tracking (MOT) [3,16]. 

Multiple Object Tracking is when different objects are tracked at the same 
time within the same video or the same set of frames [17]. For example, you 
have a video of a meal preparation activity, and you want to track the location 
of the pans, spoons, and utensils continuously throughout the video in real-time 
by estimating their trajectory. The proposed real-time multiple object tracking 
(MOT) systems typically follow the tracking-by-detection paradigm [5,12]. It 
includes (1) a detection model for the target location and (2) an appearance 
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embedding model for data association. In other words, the proposed MOT app- 
roach has to accurately detect the objects in each frame and provide a consistent 
label for them. There are certain cases where the visual appearance of the object 
is not clear as long as there is a moving object in the food preparation activity. 
For example, a spoon may be hidden behind a frying pan. In all such cases, 
detection would fail while tracking would succeed. That’s why we combine these 
two modes to provide a safe environment for food preparation. 

Detecting and localizing the object in a video in a fraction of a second and 
with high accuracy is expected and required from the proposed MOT algorithm. 
Due to the variety of stove burners and backgrounds with similar colors or tex- 
tures as the object, this detection speed can be significantly affected. Another 
significant difficulty in the context of meal preparation is occlusion. This is when 
the object in question is partially or completely occluded by another object. A 
further problem is that an object in a video can have different sizes and direc- 
tions. In food preparation, there are often two objects crossing each other. How 
do you know which is which? This problem is known as identity switching. 
Another problem with MOT, which is also important in detecting and recogniz- 
ing objects, is called Motion Blur. Motion blur is when the object is blurred due 
to the motion of the object or the motion of the camera. As a result, the object 
no longer looks the same. A quick move of the hand in front of the camera to 
grab an object on the cooking surface or to turn the soup in the pot. 

Two object tracking techniques are used to solve these problems: Detection- 
Based Tracking and Detection-Free Tracking [20]. In detection-based tracking, 
successive video frames are presented to a pre-trained object detector. The detec- 
tor generates detection hypotheses that are used to build tracking trajectories. 
In detection-free tracking, a fixed number of objects must be manually initialized 
in the first frame. In subsequent frames, it then tracks these objects. It is not 
able to deal with the case where new objects appear in the intermediate frames. 

Typically, objects move back and forth during meal preparation. For our 
context, this means that non-tracking detection is not appropriate. In addition, 
new objects appearing and disappearing must be automatically supported, i.e., 
recognized when they appear and forgotten when they disappear. Therefore, 
tracking-based detection could meet our needs. Finally, the tracker is used to 
make the remaining predictions, and the object detector is run for every n frames. 
For tracking over a long period of time, such as food preparation, this approach 
is very suitable. 


3.1 General Algorithm 


To track objects appearing on the cooking surface of the stove during a meal 
preparation activity in real-time, this study proposes a hybrid method. For this 
purpose, the Kernelized Correlation Filter (KCF) [8], which is particularly fast 
in tracking, is used in combination with the YOLO [15], which is an accurate 
and relatively fast detection model. The workflow of the proposed method is 
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shown in Fig. 2. First, the objects in the video image were detected by the 
YOLO algorithm, and the surface target was continuously tracked by the KCF 
algorithm. After tracking a certain number of frames, the detection mechanism 
is re-introduced, and then the output of the target frame is obtained through 
the feature fusion algorithm based on machine learning, and when new targets 
appear in the field of view, the new targets are initialized and tracked. 


Object 
Input : Frame detection No Den ene 
YOLO 


Yes 


Tracked but Update tracked 
undetected object with detected 


object object position 
Draw 
Ken Medial bounding box 
Yes 
No 


Update tracked object 
position with tracking Remove from and tracked 
model (KCF) the tracked objects 


object 


Fig. 2. General Algorithm for Safe COOK using YOLO and KCF. 


To summarize, an input in the form of a video file is processed in the pipeline, 
and the output is the bounding box of each object related to food preparation 
detected in the video frame, with its identity obtained from the tracking process, 
with the possibility of automatically adding an object or stopping tracking it. 
Two main concepts are proposed: (1) estimation model; and (2) data mapping. 
The estimation model is used to estimate the next bounding box of a detected 
bounding box, while the data mapping is used to map the predicted next bound- 
ing boxes to the actual or detected bounding boxes to link them into a unique 
identification number. The KCF filter must be used to update and predict the 
data with the equations described for each step to complete the tracking process. 
The value is a measurement from each frame. In this case, it is represented as 
the center of gravity of the bounding box found by YOLO. This process must be 
performed along the entire video stream, with each frame considered a different 
state and the predicted values updated recursively with the measurements. 
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3.2 YOLO Detector Model 


This algorithm is called You Only Look Once (YOLO) because it identifies 
objects and their locations using bounding boxes by looking at the image only 
once [15]. YOLO is proposed for real-time object detection in high-speed video 
using a convolutional neural network architecture. It has several advantages over 
traditional methods, including high speed, low computational costs, and fewer 
background errors. Instead of processing each class separately, the YOLO net- 
work turns the detection problem into a regression problem where the image is 
used only once as the input network. This makes it much faster than traditional 
object detection algorithms to determine the location of detected objects in the 
video frames. The single deep convolutional neural network used by YOLO is 
capable of predicting all the bounding boxes and the class probabilities for those 
boxes at the same time, using features from the entire image. It is suitable for 
end-to-end training in real-time while maintaining good average accuracy. The 
YOLO object detector is fast because it works in one step. The image is first 
divided into an SxS grid; each grid cell predicts B bounding boxes, including 
the coordinates, the width, the height, the confidence values of the box, and the 
conditional class probabilities. 


3.3 Kernelized Correlation Filter Tracker Model 


Correlation measures how similar two patterns are, with more similar patterns 
being more highly correlated. It is obtained by the pairwise product of corre- 
sponding pixels. Correlation filtering, which uses trained sample images that 
capture the appearance of objects of interest, is a widely used method in visual 
tracking applications. Unfortunately, the large sample size requirement of cor- 
relation filters becomes computationally demanding, conflicting with real-time 
requirements. However, the limitation of the number of samples can be at the 
expense of performance. A novel idea that significantly increases the tracker’s 
computational speed is to exploit the circulant structure of the data matrix for 
larger sample sizes and to extend it using a kernel trick. This is known as the Ker- 
nelized Correlation Filter (KCF) tracker [8]. To perform fast tracking, the KCF 
reveals the relationship between the circulant matrix and the discrete Fourier 
transform. The circulant matrix rows consist of the target model and the model’s 
cyclic shifts. The ridge regression used to learn the image windows is defined in 
the frequency domain using the Discrete Fourier Transform. There are two main 
phases, training, and detection, in the KCF tracking algorithm. In the training 
phase, there is a training of the classifier based on a set of samples. Three- 
dimensional color or oriented gradient histograms are used as feature vectors in 
this phase. In the detection phase, a new sample is detected by the operation of 
the Fourier transform. Typically, this involves multiplying the weights obtained 
after training by the frequency domain test images to find the possible position 
of the target. Before any computation can be performed, each tracker requires 
the extraction of frames. Once the frames have been extracted, the KCF needs 
a training model with the image patch at the initial position of the target. 
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4 Multi-object Tracking in COOK 


In this study, we propose a fast object-tracking pipeline that combines accu- 
rate YOLO detection and fast KCF tracking to track specific objects for the 
preparation of meals. Typically, several objects appear and disappear quickly on 
the cooking surface during a meal preparation activity. For this reason, we have 
added an extra step to perform a fast drop tracking. This solution was chosen 
because it can be useful to work in an environment where resources are limited, 
such as in an embedded system. The overall architecture of the implemented sys- 
tem is shown in Fig. 3. The system uses deep learning, OpenCV, and Python to 
detect objects in images and video streams using the YOLO object detector. We 
created an additional dataset of images with different types of kitchen utensils 
to train the YOLO models and test the proposed methods. 
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Fig. 3. Architecture of the Safe COOK system, showing how the MOT can provide 
more information in the context of hazardous situation detection. 


In this architecture, the camera collects data from the stove and uses the 
ZeroMQ (ZMQ) protocol to transfer the data. ZeroMQ is a high-performance 
asynchronous messaging library intended for use in distributed or concurrent 
applications. This information is then sent to OpenCV, which runs the proposed 
solution for multiple object detection. The proposed real-time method, through 
an object detection, localization, and correction process, automatically detects 
and tracks the utensil appearing in a sequence of images. The central coordinates 
(x, y) of YOLO are updated every thirty frames. This is a trade-off between 
accuracy and speed. 

A home kitchen with a stainless steel countertop and a spiral kitchen lay- 
out was used to record the meal preparation videos. The videos were recorded 
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Fig. 4. Sample frames from our training dataset. Showing detection of hands, stoves, 
pans, and spoons. 


with a high-resolution camera. The camera was connected directly to a 4 GB 
Raspberry PI 4 model B+. Image transport was done using imageZMQ, a set of 
Python classes that transport OpenCV images between computers using PyZMQ 
messaging. Experiments were done for the YOLOv5 network. Existing data like 
OpenImage provides a large set of trained and annotated images (bounding box) 
with 600 classes. However, this number of classes is limited, and some classes do 
not have enough images to do the training properly. 

Therefore, data from OpenImage and Google Colab were used for training on 
an IntelR, Core TM i7-3770 CPU 3.40GHz 4 cores with 8 threads and Raspberry 
Pi 4 with 4 GB. Mainly we labeled the following classes: Pan, Hand, Stove, and 
Spoon as shown in Fig.4. There are many datasets for testing object tracking 
methods in videos. However, these datasets do not consist of meal preparation 
videos. Therefore, to create a dataset for our specific safe cook app that tracks 
meal preparation, we tagged 3500 frames. From this dataset, 1000 frames are 
used for the training of the YOLOv5 models, while the remaining 2500 frames 
are used for the testing. 


5 Results and Discussion 


This solution adds further context to the detection of hazardous situations. It 
provides data that is combined with sensor information to offer greater safety 
during meal preparation. However, some problems need to be solved to make it 
more optimal. 

Foremost, YOLO does not recognize the uniqueness of certain objects from 
one image to the next. For example, there is no way to be sure that the detected 
pan is the same pan that was detected in the previous image. Our solution is to 
compare the position of objects detected by YOLO with the position of objects 
detected by tracking. So if we track an object of the same type at the position of 
the YOLO detected object (with some error depending on the implementation), 
we consider them the same objects. With this approach, we are now able to 
recognize the type and the instance of the objects. For example, we will be able 
to distinguish between panl, pan2, and so on. 
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The second problem is that it is possible for the tracking to start tracking a 
similar object instead of the previously detected object. This happens even when 
using KCF, which was specifically chosen for its ability to easily stop tracking 
when the object is lost. This is most likely to happen with objects that are in 
proximity to identical objects. For example, how can you track an object that 
has the same color as the image background? How can you be sure that you still 
have the right object on the track? The solution chosen for this problem is to 
use object detection (YOLO) with a very low confidence level. 


Fig. 5. Performance for tracking and detecting objects in interaction 


The objects detected with this confidence level are compared to the tracked 
objects. In other words, does the object on the track have a minimal chance of 
being the object on the track? If this is not the case, then we will stop tracking 
this object. Since the tracking is usually stopped because of this condition and 
not because of the loss of tracking, the overall result is a detection algorithm 
that makes it easier to detect objects (minimum confidence level), but at the 
same time correctly identifies objects (higher initial confidence level). On the 
other hand, the effectiveness of monitoring is greatly reduced with this solution. 
Therefore, it is ideal to find another solution that would easily drop tracking. 

The object detection and tracking techniques of YOLO and KCF can be 
easily substituted or replaced for different models due to the modularity of our 
algorithm. We replaced KCF with other tracking algorithms to evaluate the effec- 
tiveness of our tracking loss algorithm. Starting from the simple use of YOLO 
without tracking, then YOLO with single tracking without detecting multiple 
instances of objects, then YOLO with tracking of multiple instances of objects, 
and finally YOLO with tracking of multiple instances of objects with drop track- 
ing. The sequences obtained are shown in Fig. 5. 
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Table 1. Performance evaluation of the different methods with and without tracking. 


Frame Per Objects Objects 
seconds detected detected 
Pipeline (Average) | (Per image) | (Per seconds) | Comments 


TT Basily loses objects from = 
frame to frame, 

YOLO 3.71 0.99 3.67 especially when objects overlap. 

Can only track an object 


of the same type, and may 

YOLO + SOT 1.96 1.83 3.57 start tracking the wrong object. 

May start tracking the wrong object. 
On the other hand, has the best 

YOLO + MOT 1.81 3.81 6.90 follow-up when he is not mistaken. 

Most consistent tracking and low 


chance of tracking a bad object. 
YOLO + MOT + Less effective than no drop in the 
Drop tracking 2.49 1.28 3.19 ideal situation. 


Note that the multiple tracking algorithms without drop tracking have the 
best performance metric. However, it often tracks the wrong objects. So on 
the whole it is not optimal. In addition, although YOLO alone has a better 
performance metric than the full algorithm, the full algorithm (MOT-+drop) 
has better detection and is still able to do detection at sufficient speed (real- 
time) to be useful in the context of the Safe Cook project. The performance 
quantified on a video of the experiment is shown in Table 1. 


6 Conclusion 


SafeCOOK is a real-time system for tracking multiple objects associated with 
preparing a meal while using the Cognitive Orthosis for CoOKing (COOK). It 
provides more contextual information to the safety self-monitoring system for the 
detection of potentially dangerous situations that are not currently managed. 
The proposed system is a low-cost monitoring system without a graphics 
card based on YOLOv5 and KCF. The main idea is to reduce the limitations 
of the resulting tracking system by combining a tracker and a detector to take 
advantage of both methods. In order to meet the requirements of the fast and 
frequent appearance and disappearance of cooking utensils on the cooking sur- 
face, we propose in this paper an algorithm that fuses the detection, the object 
tracking, and the fast object dropout tracking. The fusion algorithm is designed 
in such a way that the detection and tracking information is complementary in 
order to fulfill the tasks of detecting hazardous situations during food preparation 
in different scenarios. The findings show that the detection and tracking fusion 
algorithm can effectively detect and track kitchen tools in different scenarios, 
and the success rate exceeds that of a single detection and tracking algorithm. 
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Abstract. The end of the covid-19 epidemic has revealed many weak- 
nesses in the health system and the medical waste treatment process. 
In particular, the ineffective treatment of medical waste has also con- 
tributed to the explosion in the number of infections in some countries 
(i.e., India, Brazil, and Vietnam). Several studies have found that even 
developed countries (i.e., with better infrastructure and health services 
than the world average) have to face emergencies during this time. epi- 
demic. Therefore, the amount of medical waste dumped into the envi- 
ronment is extremely terrible. The waste generated suddenly during 
this period includes protective gear, masks, and vaccines that burden 
the waste treatment process. There are several approaches to exploiting 
Blockchain technologies to solve the problem of direct contact between 
the stages: medical staff - transportation staff - waste disposal staff to 
minimize unintended spread. However, to thoroughly solve the current 
waste classification and treatment processes, a more reward/punishment 
solution is needed. Specifically, we propose a model to assess the com- 
pliance/violation level of waste sorting and treatment in medical centers 
and isolation areas based on current popular technologies: blockchain, 
smart contracts, and NFTs. 


Keywords: Medical waste treatment - Blockchain - Smart contracts - 
NFT - Ethereum - Fantom - Polygon - Binance Smart Chain 


1 Introduction 


The passing of the Covid epidemic has revealed inadequacies in the treatment 
of medical waste. Studies conducted by [10] have shown that medical waste 
directly contributes to the spread of Covid disease. Other studies have also shown 
that upgrading medical facilities (i.e., waste disposal processes) is not enough to 
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ensure a safe, closed process in a healthcare environment. Therefore, a series of 
approaches have been born [1,17] to speed up the processing process to support 
medical doctors and e-medical staff, where direct contact is minimized (ie., face- 
to-face). These solutions also support the doctor/nurse-patient visit process. 
Specifically, patients will report information online about their symptoms. The 
doctors will give the corresponding advice to help the patient have an effective 
treatment plan. 

However, for the medical waste treatment process, where there is a need 
for interaction between many stakeholders (i.e., nurses, doctors, patients, staff 
in medical center departments, as well as employees of waste transporta- 
tion/disposal companies). Indeed, 99% of items (including medical equipment 
and supplies) become junk after use within the first six months of first use [13]. 
Therefore, many approaches [3,5] have presented processes based on central pro- 
cessing in the waste treatment process before the time of the Covid epidemic. 
However, these solutions do not take into account the risk reduction of direct 
contact between stakeholders (eg, health workers - transportation companies or 
transport companies - waste treatment companies). In addition, the approach 
based on traditional centralized data storage faces many risks, such as a lack of 
transparency between information transfer stages [9]. Centralized storage also 
prevents information authentication between the parties involved in the tradi- 
tional waste treatment process [6,7]. In addition, information is easily hacked 
or stolen by malicious users [20] because all information is stored centrally and 
is difficult to recover when attacked by hackers or natural disasters (eg, fire or 
flood). 

Because of the above risks, many approaches exploit Blockchain technology 
as an alternative to centralized storage, and lack of transparency [14]. In the mat- 
ter of waste treatment, the majority of systems deployed on two main platforms, 
Ethereum [8] and Hyperledger [12] offer solutions medical waste treatment (see 
Related work for details). However, these approaches have not yet provided a 
specific definition for reward behaviors and violations for individuals and orga- 
nizations for their waste sorting behavior. 

Therefore, in this article, we propose a medical waste classification model 
based on the combination of a series of current advanced technologies including: 
Blockchain, smart contract, and NFT (see Approach section for more details). 
In addition, to demonstrate the effectiveness of the proposed model we imple- 
mented a proof-of-concept based on the Ethereum platform (i.e., see 2 section 
for reasons to use this platform instead of other platforms) combined with ERC 
721 (i.e., Ethereum’s NFT issuance certificate). We also install smart contracts 
on three EVM (Ethereum Virtual Machine) enabled platforms including BNB 
smart chain, Fantom, Polygon and Celo (i.e., see Evaluation for more details). 

Therefore, our work contributes on four aspects. (a) Propose a mechanism for 
classification and treatment of medical waste based on blockchain technology and 
smart contract; (b) propose a model to create reward/punishment decisions for 
individuals/organizations based on NFT technology; (c) implement the proposed 
model based on smart contracts and the proposed model (i.e., proof-of-concept); 
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and (d) deploying proof-of-concept on 4 supporting platforms (ERC721 - NFT 
of ETH) and EVM (deploying smart contract implemented in solidity language) 
including BNB Smart Chain, Fantom, Polygon, and Celo.! 


2 Related Work 


2.1 Ethereum-Based Waste Treatment System 


Two popular examples in applying the Ethereum platform in building a supply 
chain management model for waste treatment in the healthcare environment are 
[8]. For electronic medical devices (eg, CT scanner, X-Ray machine, pacemaker, 
etc) that are no longer in use (e.g., damaged, expired) at the centers healthcare 
is called e-waste, Gupta et al. [8] proposes a system to treat the above types of 
waste based on Blockchain technology (i.e., Ethereum). To develop processing 
policies that comply with strict requirements at medical centers, the authors have 
built a system of smart contracts. In order to raise awareness among users about 
waste disposal, each user will receive a reward if they complete the sorting before 
sending them for disposal. However, these solutions only apply smart contracts 
to distribute reward points instead of storing evidence of reward receipts or 
violations in the waste disposal process - as we have applied (i.e., based on 
NFT). 

Another model was introduced by Laura et al. [11], which is implemented on 
the Ethereum platform. This system helps observers in assessing violations in 
solid waste segregation (computers and smartphones). The difference between 
this paper and other approaches is that it proposes a remote management model 
based on QR code (i.e., generated from classification to processing). For cross- 
border processes (i.e., between countries rather than within the same country) 
Schmelz et al. [16] introduces an Ethereum-based system, privacy protection 
model, for garbage disposal. Information that is not related to the waste treat- 
ment process will be stored offchain instead of putting them on-chain. This 
approach is similar to our NFT-based certificate and metadata storage model. 
However, our focus is on encouraging standard behavior in waste sorting as well 
as violations of the above procedure. 


2.2 Waste Treatment System Based on Hyperledger Fabric 


In this section, we show three case studies in applying the Hyperledger platform 
to propose a medical waste treatment model. Most waste products during the 
Covid 19 pandemic are listed as hazardous waste to health (i.e., 99% of items 
after use are medical waste [13]). So they combine blockchain and smart contracts 
to classify and dispose of garbage instead of just centralized local storage. For 
example, Trieu et al. [12] proposes a waste treatment model called MedicalWast- 
Chain. This proposed model is aimed at the treatment of medical waste from 


1 We do not deploy smart contracts on ETH because the execution fee of smart con- 
tracts is too high. 
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medical centers, reuse of tools, the process of transferring medical supplies waste 
(i.e., protective gear, gloves, masks, etc.) and waste treatment processes in fac- 
tories. They aim for a solution to help parties trace the source of waste and 
toxic levels during the pandemic. Similar to the above approach, Ahmad et al. 
[2] aims at the traceability model of personal protective equipment for health- 
care workers (i.e., doctors, nurses, testers) during the pandemic. To assist with 
validation of waste treatment processes (i.e., stakeholder interactions), Dasaklis 
et al. [4] proposed a blockchain-based system set up on smartphones. 

However, the above approaches do not pay too much attention to the regener- 
ation/refurbishment process. Neither of the above approaches offers a reasonable 
solution for handling (i.e., reward and handling of violations). Specifically, the 
above solutions (for both Hyperledger Eco-system and Ethereum platforms) only 
focus on the management model of the waste treatment chain from the place of 
origin (i.e., medical centers) to factories. In this paper, we not only offer a model 
to manage the waste sorting process, but also provide a solution for rewarding 
and handling user violations based on NFT technology. 


3 Approach 


3.1 The Traditional Model of Medical Waste Treatment 


Recycling 
medical product 


fs 
fela El = w 1 
-2a 
Medical center Medical Self 


equipment & supply Recycle Medical waste 
Js 


~ 
5 tO = 


== [VVU] 


Recycling Transportation Sorting 
plant factory 


Fig. 1. The traditional model of treatment and classification of medical waste 


To build a model of treatment and classification of medical waste that is com- 
monly applied today, we refer to the process of classifying and treating medical 
waste during the Covid-19 pandemic. In the documents that can be downloaded 
and referenced from CDCs in different parts of the world, we choose Vietnam 
for two reasons: i) Vietnam has successfully controlled through the first two 
waves of Covid-19 in 2019 and early 2020; ii) is a developing country - undevel- 
oped waste treatment infrastructure can be applied in other countries around 
the world. Specifically, in contrast to complicated waste treatment processes 
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- difficult to apply to other countries due to budget constraints and modern 
equipment - the application model in Vietnam can be applied. in other countries 
around the world. The process of classification and treatment of medical waste 
during the Covid-19 epidemic was signed by the Ministry of Health in 2019 [15]. 
The steps are shown in Fig. 2. Specifically, Fig. 2 shows five sources of medical 
waste classification and five treatment steps. Sources of medical waste include 
treatment place (i.e., hospital, military barracks), testing, vaccination, and the 
personal place under quarantine (eg,household, apartment). For medical waste 
classification, the first three steps are carried out at healthcare centers (i.e., 
classification, separation, and collection) where all hazardous waste is sent to 
factories for treatment disposal (i.e., destruction) including transportation and 
treatment. 
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Fig. 2. The Covide-19 pandemic-related medical waste treatment sample in Vietnam 


Figure 1 shows the steps of traditional garbage classification and treatment. 
Step 1 presents the waste collection process in departments in medical facilities 
(e.g., hospitals). These types of waste include medical equipment and supply. Step 
2 presents the process of self-segmentation and reuse in the medical environment. 
At the end of this process, products belonging to the recycling medical product 
group are reused (step 3) while medical waste is disposed of (step 4). All medical 
waste is sent to the waste treatment area (step 5). Here, depending on the treat- 
ment requirements, the medical waste is classified into the corresponding treat- 
ment processes (i.e., recycling plant and sorting factory - step 6). 

The risks in the traditional medical waste treatment process have been sum- 
marized and presented in the Introduction and Related work sections. However, 
in order to improve people’s sense of self-segregation and treatment of medical 
waste, we propose a model that combines blockchain, smart contract, and NFT 
technologies to create certificates in classification at medical centers thereby 
identifying compliance /violation. 


210 H. Q. Nguyen et al. 


3.2 Model of Treatment and Classification of Medical Waste Based 
on Blockchain Technology, Smart Contract and NFT 
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Fig. 3. Model of treatment and classification of medical waste based on blockchain 
technology, smart contract and NFT 


Figure 3 presents nine steps of classification and treatment of medical waste using 
blockchain technology, smart contract and NFT. In particular, doctors, nurses, 
and other medical center staff (eg, clinics, hospitals) must confirm constraints on 
waste segregation at the workplace (step 1). These requirements are presented 
in the form of guidelines and commitments (i.e., specify compliance/violations 
when sorting medical waste). Here, we only classify into two main groups includ- 
ing: recyclable waste at medical centers and hazardous waste. Step 2 presents 
healthcare workers with self-segregation of waste as required in step 1. Hazardous 
items must be isolated from the treatment and medical care area (i.e., waiting 
for the transporter to arrive at the centers). waste treatment center). In step 
4, hospitals provide guidance documents to cleaning staff regarding the assess- 
ment of compliance/violation behaviors by doctors and nurses. Step 5, present 
the cleaning staff’s monitoring process for all types of waste after being sorted 
(i.e., check the process for hazardous waste). Step 6 shows the cleaning staff 
to check the types of recyclable waste. These assertions are sent to the respec- 
tive functions (i.e., name the functions in the smart contract) - step 7. Step 
8 generates NFTs corresponding to compliance/violations based on cleaning’s 
evaluation process. staff for individuals/organizations at medical centers. The 
entire process is stored in the distributed ledger (step 9). 


4 Implementation 


Our reality model focuses on two main purposes: i) data manipulation (i.e., 
medical waste) - initialization, query and update - on blockchain platform and 
ii) creation of NFTs for each user’s (i.e., individual/organization) reward and 
violation behavior based on their behavior in waste sorting/disposal. 
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4.1 Initialize Data/NFT 
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Fig. 4. Initialize data/NFT 


Figure 4 shows the steps to initialize medical waste data. These types of waste 
include medical equipment (i.e., expired/damaged) or medical supplies (eg, 
masks, PPE, injections). These types of waste are required to be classified into 
different categories (i.e., discard, reuse) depending on their level of toxicity. Then 
descriptions of the type of waste are added to each specific garbage bag. Each 
trash bag has a unique address to separate them with the type of waste. In 
addition, information about the sorter as well as in which department, time, 
and location is also added to the metadata of the trash bag. As for the stor- 
age process, services support concurrent storage (i.e., distributed processing as 
a peer-to-peer network) on a distributed ledger - supporting more than one user 
for concurrent storage to reduce system latency. The medical waste data is orga- 
nized as follows: 


medicalWasteData0bject = { 
"medicalWasteID": medicalWastelID, 
"staffID": staffID, 

"type": type of waste, 
"apartmentID": apartmentID, 
"quantity": quantiy, 
"unit": unit, 

"packageID": packagelID, 
"time": time, 

"location": location, 
"state": null, 

"reUse": Null}; 


Specifically, in addition to the information to extract the content (i.e., place 
of origin, weight, type of waste, etc.), we also store information related to the 
status of the garbage bags at the hospital (ie. , “state” and “reUse” - defaults to 
Null). Specifically, “state” changes to 1 if the corresponding garbage bag has been 
shipped out of the medical center (i.e., for the type of waste to be treated); value 
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0 - pending. Meanwhile, “reUse” presents the value 1 when the type of waste 
(i.e., medical device) is reused (i.e., value 0 - pending). Non-hazardous wastes 
(i.e., non-toxic to the environment and user’s health). After the waste sorting 
phase, the cleaning staff will check if they are in accordance with the process and 
wait for validation before synchronizing on the chain (i.e., temporarily stored on 
the data warehouse). Then the pre-designed constraints in the Smart Contract 
are called through the API (i.e., name of function) to sync them up the chain. 
This role of accreditation is extremely important because they directly affect 
the waste treatment process, as well as the premise for rewarding or sanctioning 
individuals and organizations. 


4.2 Data Query 


Fig. 5. Data quer Fig. 6. Data updated 
8 y 8 


Similar to the data initialization steps, the data query process also supports 
many simultaneous participants in the system for access (i.e., distributed model). 
Support services receive requests from Cleaning staff or nurses/doctors to access 
data. Depending on the query object we have different access purposes. Specifi- 
cally, Cleaning staff queries with the purpose of checking the classification pro- 
cess or transferring hazardous medical waste to the shipping company. In con- 
trast, healthcare professionals can query data to find reusable medical tools 
(i.e., out of stock). Figure 5 shows the steps to query medical waste data. These 
requests are sent as requests (i.e., pre-designed services as API calls) from the 
user to the smart contracts available in the system (i.e., name of function) before 
retrieving the data. from the distributed ledger. All retrieval requests are also 
saved as query history for each individual or organization. In case the correspond- 
ing information is not found (eg, wrong ID), the system will send a message not 
found results. For the NFT query process, all support services are provided as 
APIs. 


4.3 Data Updated 


The data update routine is invoked only after verifying that the data exists on 
the thread (i.e., after executing the corresponding data query procedure). In this 
section, we assume that the search data exists on the string. Where none exists, 
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the system sends the same message to the user (see Sect. 4.2 for details). Similar 
to the two processes of query and data initialization, we support update services 
in the form of APIs to receive requests from users before passing them to smart 
contracts (i.e., name of function) for processing. The purpose of this process is 
to update the time and location of the garbage bags during transportation and 
handling of medical waste. Thereby, the administrator can trace the status of 
medical waste treatment /transportation from medical centers to waste treatment 
companies. Figure6 shows the procedure for updating medical waste data. For 
NFTs (i.e., available) the update process includes only moving from the owner’s 
address to the new (i.e., new owner). If any information is updated on an existing 
NFT, it will be stored as a new NFT (see Sect. 4.1 for details). 


5 Evaluation 


Since the proposed model rewards/penalizes for compliance/violation of the 
medical waste classification process, we implement the recommendation model 
on EVM-enabled blockchain platforms instead of mining platforms under the 
Hyperledger eco-system. In addition, assessments based on system responsive- 
ness (i.e., number of requests responded successfully /failed, system latency - min, 
max, average) have been evaluated by us in the previous research paper. There- 
fore, in this paper we determine the suitable platform for our proposed model. 
Specifically, we install a recommendation system on four popular blockchain plat- 
forms today, supporting Ethereum Virtual Machine (EVM), including Binance 
Smart Chain (BNB Smart Chain)?; Polygon*; Fantom*; and Celoř. Our imple- 
mentations on these four platforms are also shared as a contribution to the 
article to collect transaction fees corresponding to the four platforms’ support- 
ing coins®, ie, BNB”; MATIC®; FTM®; and CELO!°. Our implementations to 
evaluate the execution cost of smart contracts (i.e., designed based on Solid- 
ity language) run on testnet environments of four platforms in order to choose 
the most cost-effective platform to deploy. Our detailed assessments focus on the 
cost of performing contract creation, NFT generation and NFT retrieval/transfer 
(i.e., updating NFT ownership address) presented in the respective subsections 
related to i) Transaction Fee; ii) Gas limit. 


? https: //github.com/bnb-chain/ whitepaper /blob/master/WHITEPAPER.md. 

3 https: //polygon.technology /lightpaper-polygon.pdf. 

* https: //whitepaper.io/document /438/fantom-whitepaper. 

5 https: //celo.org/papers/whitepaper. 

€ Implementation of theme models our release at Nov-24-2022 04:19:05 AM +UTC. 

T https: //testnet.bscscan.com/address/0x94d93a5606bd3ac9ae8b80e334dfec74d0075e 
ce. 

8 https: //mumbai.polygonscan.com/address/0x48493a3bb4e7cb42269062957bd541d5 
2afc0d7a. 

? https://testnet.ftmscan.com/address/0x48493a3bb4e7cb42269062957bd541d52afc0 
d7a. 

10 https: //explorer.celo.org/alfajores /address/0x48493A 3bB4E7cB42269062957Bd541 

D52aFc0d7A/transactions. 
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Table 1. Transaction fee 


Contract Creation Create NFT Transfer NFT 

BNB Smart Chain | 0.02731376 BNB ($8.41) 0.00109162 BNB ($0.34) 0.00057003 BNB ($0.18) 

Fantom 0.009577666 FTM 0.000405167 FTM 0.0002380105 FTM 
($0.001840) ($0.000078) ($0.000046) 

Polygon 0.006841 190030101236 0.000289405001041858 0.000170007500612027 
MATIC($0.01) MATIC($0.00) MATIC($0.00) 

Celo 0.0070979376 CELO ($0.004) 0.0002840812 CELO ($0.000) 0.0001554878 CELO ($0.000) 


5.1 Transaction Fee 


Table 1 shows the cost of creating contracts for the four platforms. It is easy to see 
that the highest transaction fee of the three requirements is contract creation for 
all four platforms. In which, the cost of BNB Smart Chain is the highest with the 
highest cost when creating a contract is 0.02731376 BNB ($8.41); whereas, the 
lowest cost recorded by the Fantom platform with the highest cost for contract 
initiation is less than 0.009577666 FTM ($0.001840). Meanwhile, the cost to 
enforce Celo’s contract initiation requirement is lower than Polygon’s with only 
$0.004 compared to $0.01. For the remaining two requirements (Create NFT 
and Transfer NFT), we note that the cost of implementing them for all three 
platforms, Polygon, Celo, and Fantom is very low (i.e., negligible) given the 
cost. trades close to $0.00. However, this cost is still very high when deployed on 
BNB Smart Chain with 0.00109162 BNB ($0.34) and 0.00057003 BNB ($0.18) 
for Create NFT and Transfer NFT, respectively. 


5.2 Gas Limit 


Table 2. Gas limit 


Contract Creation | Create NFT | Transfer NFT 
BNB Smart Chain | 2,731,376 109,162 72,003 
Fantom 2,736,476 115,762 72,803 
Polygon 2,736,476 115,762 72,803 
Celo 3,548,968 142,040 85,673 


Table 2 shows the gas limit for each transaction. Our observations show that the 
gas limits of the three platforms (i.e., BNB, Polygon, and Fantom) are roughly 
equivalent - where Polygon and Fantom are similar in all three transactions. The 
remaining platform (i.e., Celo) has the highest gas limit with 3,548,968; 142,040; 
and 85,673 for all three transaction types. 


5.3 Discussion 


According to our observation, the transaction value depends on the market 
capitalization of the respective coin. The total market capitalization of the 4 
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platforms used in our review (i.e., BNB (Binance Smart Chain); MATIC (Poly- 
gon); FTM (Fantom); and CELO (Celo)) are $50,959,673,206; $7,652,386,190; 
$486,510,485; and $244,775,762.'! This directly affects the coin value of that 
platform - although the number of coins issued at the time of system imple- 
mentation also plays a huge role. The coin’s value is conventionally based on 
the number of coins issued and the total market capitalization with a value of 
$314.98; $0.863099; $0.1909; and $0.528049 for BNB, MATIC, FTM, and CELO 
respectively. Based on the measurements and analysis, we have concluded that 
the proposed model deployed on Faltom brings many benefits related to sys- 
tem operating costs. In particular, generating and receiving NFTs has an almost 
zero (i.e., negligible) fee. Also, the cost of creating contracts with transaction 
execution value is also meager (i.e., less than $0.002). 

In future work, we proceed to implement more complex methods/algorithms 
(i.e., encryption and decryption) as well as more complex data structures to 
observe the costs for the respective transactions. Deploying the proposed model 
in a real environment is also a possible approach (i.e., implementing the recom- 
mendation system on the FTM mainnet). Currently, we have not considered issues 
related to the privacy policy of users (i.e., access control [18], dynamic policy [19]) 
- a possible approach would be implemented in upcoming research activities. 


6 Conclusion 


Our work proposes a reward/punishment model for individual/organizational 
medical waste sorting behavior. Our proposal is based on blockchain technology, 
smart contracts (i.e., transparency, distributed storage, and processing), and NFT 
(i.e., immutable) to build waste separation and treatment processes. medical. We 
have implemented proof-of-concept on the Ethereum platform and smart con- 
tracts (i.e., contract creation, NFT create, NFT transfer) based on the Solidity 
language. We also tested smart contracts on four EVM-enabled platforms (i.e., 
BNB, MATIC, FTM, and CELO) to find the right platform (i.e., the lowest smart 
contract execution cost). Our analysis in the evaluation section demonstrated that 
implementations on Fantom offer more benefits than the other ones. 
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Abstract. This study aimed to quantitatively analyze pressure parameters in dif- 
ferent high-risk areas depending on the position. We reviewed the clinical records 
of trials of 20 healthy adults on a multi-actuated bed accompanied with pressure 
sensor mat. We collected average, maximal, minimal pressure, and area in the 
supine and bilateral side-tilt positions. Also, we analyzed the difference between 
each at-risk area, depending on positions. In the supine position, pressure param- 
eters of the head, shoulders, sacrum, coccyx, and heels showed significant differ- 
ences, except between the right and left heels. In the right side-tilt position, all 
pressure measurements of the ear, shoulder, elbow, hip, knee, and lateral ankle were 
significantly different. In the left side-tilt position, most of the pressure parameters 
of the ear, shoulder, elbow, hip, knee, and lateral ankle were significantly different, 
except between the elbow and ankle. We found that frequent position changing is 
more important than achieving optimum positioning. 


Keywords: Patient care - Patient positioning - Pressure ulcer 


1 Introduction 


Pressure ulcers are skin and/or underlying tissue lesions caused by the limitation of 
blood circulation owing to the combination of increased pressure, friction, and sheer 
force on specific parts of the body [1, 2]. These pressure injuries are closely related to 
loss of quality of life, impairment in mobility, and increased mortality, both directly and 
indirectly [3]. 

Repositioning patients every 2 h (q2hr) to prevent continuous high pressure on a 
specific body part is the mainstream of pressure sore care, and it has been considered 
a guideline for general patient care [4, 5]. However, previous studies have shown that 
standard turning does not sufficiently relieve the high skin-bed pressure on peri-sacral 
areas, such as the sacrum, coccyx, and ischial tuberosity, even in non-disabled adult 
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subjects. This may explain why pressure ulcers still develop despite preventive methods, 
including regular patient repositioning [6]. In addition, recording repositioning has not 
been standardized or automated, and remains manual. This causes a lack of optimal 
repositioning and incomplete documentation of a patient’s position. 

Several studies have focused on a preventive solution for pressure ulcers with real- 
time position monitoring systems. In particular, devices that use sensors to detect pres- 
sure are considered to be one of the most effective methods because they minimize 
care-giver’s effort [7]. It is also important because there has been issues about the pres- 
sure monitoring while sleeping. Furthermore, these devices used to modify care plans 
while preventing the development of pressure ulcers [8]. Comprehensive understanding 
of the monitoring information gathered from devices is necessary to increase optimal 
patient repositioning by clinical staff. An integrated analysis is needed between pressure 
monitoring information and known positions to prevent pressure ulcers. 

There is a lack of research evaluating pressure measurements in various positions 
considering the risk areas of each position. This study aimed to clarify the key clinical 
implications for preventing pressure ulcers by quantitatively analyzing acquired pressure 
parameters in high-risk areas depending on position. 


2 Materials and Methods 


2.1 Study Design 


In this study, we retrospectively analyzed the clinical records of trials conducted on a 
multi-actuated bed with an applied pressure sensor. From August to September 2022, 
the clinical records of 20 healthy individuals were obtained. The inclusion criteria 
were: 1) over 18 years of age, 2) no underlying disease, and 3) no limitations on activ- 
ity. The reviewed data were recorded in a structured form. Additionally, we collected 
demographic data such as age, gender, height, weight, and body mass index (BMI). 

This study was approved by the Institutional Review Board of Korea University Guro 
Hospital (2022GR0463). 


2.2 Pressure Sensing with Multi-actuated Bed 


A calibrated XSENSOR IX500:256.256.22 (XSENSOR Technology Corporation, Cal- 
gary, Canada) pressure mapping system with X3 software (v8) was used. XSENSOR is 
considered the gold standard for pressure mapping and has been used in several stud- 
ies [5, 9, 10]. The XSENSOR mat consists of 65,536 sensing points, has a total area 
of 63.5 cm x 53.3 cm and a sensing area of 29.5 cm x 29.5 cm. The manufacturer’s 
specifications state that the pressure mat showed high durability in subsurface testing, 
an accuracy rate of +10% of the calibrated values, a sampling frame rate of 6.2 frames 
per second, and a spatial resolution of 1.15 mm. The device was calibrated to measure 
pressure from 0 to 200 mmHg. IP readings were transferred from the XSENSOR mat to 
a handheld monitor. 

A modern automatic bed setting (Jeong In ENS Corporation, Republic of Korea) 
was used in conjunction with the XSENSOR mat. Repositioning to the lateral tilt (i.e., 
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turning) position and supine position are the most routine methods for patients’ position 
change [11]. The multi-actuated bed supported three standard positions for the prevention 
of pressure ulcers: right-side tilt, 30°; left-side tilt, 30°; and supine [12]. 


Fig. 1. Illustration of three different position with measurement. (a) is side-tilt 30° position, (b) 
is supine position and (c) is example of pressure sensing measurements. 


In particular, the side-tilt and supine positions have different high-risk areas for the 
development of pressure ulcers. In the supine position, pressure ulcers frequently occur 
at the head (occiput), bilateral shoulders (scapula), sacrum, coccyx, and bilateral heels. 
In the lateral tilt position, pressure ulcers easily develop in the ear, shoulder (upper 
humerus), elbow outer side (lateral epicondyle), hip (greater trochanter), knee outer side 
(fibular head), and lateral ankle outer side (lateral malleolus). [13, 14] We considered 
these differences regarding positioning on the evaluations and selected the most highly 
recorded areas in each risk area. 

The sensor mat was placed above the bed, and the pressure was continuously mon- 
itored for 10 min in three positions. In total, 30 min was needed for each evaluation 
[5]. Recorded data included average pressure (N/cm2), peak pressure (N/cm2), minimal 
pressure (N/cm2), and area (cm2). The settings are illustrated in Fig. 1. 


2.3 Statistical Analysis 


Using the Wilcoxon signed-rank test, the quantitative difference between pressure param- 
eters depending on the risk areas of each position was analyzed. For the descriptive 
analysis, the median and inter-quarter range of measurements were suggested for metric 
evaluations. SPSS version 26.0 software (SPSS Inc., Chicago, IL, USA) was used for 
all analyses, with the statistical significance level set at p < 0.05. 


3 Results 


Ten patients were male and 10 were female. The median age of participants was 28 years 
(25.25-31.5); median height was 167.0 cm (162.0—172.75); median weight was 63.5 kg 
(50.25-70.0); and median BMI was 21.12 kg/m? (18.75-23.87). 


224 J. Lee and J. S. Yoon 


3.1 Pressure Measurements at Risk Area of Supine Position 


The pressure parameters in the supine position in high-risk areas are listed in Table 1. 
All four pressure measurements of the head and bilateral shoulders, right heel; coccyx 
and left heel, right heel were different. Pressure measurements of the head and left heel 
were different, except for average pressure. The pressure measurements of the sacrum 
and right heel were different, except for maximal pressure. Pressure measurements of 
the left shoulder and left heel; right shoulder and left heel; sacrum and left heel were 
different, except for minimal pressure. Pressure measurements of the left shoulder and 
sacrum, coccyx; right shoulder and sacrum, coccyx; sacrum and coccyx were different, 
except in the area. The average and minimal pressures of the left and right shoulders 
were different. The maximal pressure and areas of the head and coccyx; left shoulder and 
right heel were different. The areas of the right shoulder and right heels were different. 
There was no statistically significant differences between the right and left heels. 


Table 1. At-risk area of pressure ulcer in supine position. 


Parameters Average P Maximal P Minimal P Area (cm?) 
(N/cm?) (N/cm?) (N/cm?) 
Head (Occiput) 0.43 0.57 0.34 40.32 
(0.41-0.50) (0.53-0.60) (0.25-0.46) (40.32—40.32) 
Shoulder Rt. 0.34 0.40 0.27 90.73 
(0.32-0.37) (0.37-0.42) (0.24-0.33) (90.73-90.73) 
Lt. 0.35 0.38 0.30 90.73 
(0.31-0.36) (0.36-0.41) (0.26-0.33) (90.73-90.73) 
Sacrum 0.42 0.50 0.34 90.73 
(0.39-0.43) (0.47-0.50) (0.30-0.36) (90.73-90.73) 
Coccyx 0.47 0.50 0.41 90.73 
(0.43-0.48) (0.48-0.52) (0.37-0.44) (90.73-90.73) 
Heel Rt. 0.36 0.42 0.26 20.16 
(0.31-0.37) (0.37-0.53) (0.20-0.31) (20.16-20.16) 
Lt. 0.36 0.43 0.29 20.16 
(0.32-0.41) (0.37-0.52) (0.25-0.33) (20.16-20.16) 


Abbreviations: P, pressure; Rt., right; Lt., left 


3.2 Pressure Measurements at Risk Area of Right Side-Tilt Position 


The pressure measurements in the side-tilt position to the right in the high-risk areas are 
presented in Table 2. All four pressure measurements of the ear and shoulder, elbow, knee, 
ankle; shoulder and knee, ankle; elbow and hip; hip and knee, ankle were different. The 
pressure measurements of the ear and hip were different, except for maximal pressure. 
The pressure measurements of the shoulder and elbow were different, except for minimal 
pressure. The pressure measurements of the shoulder and hip were different, except for 
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the area. The minimal pressures and areas of the knee and ankle were different. The 
minimal pressures at the elbow and ankle were different. The elbow and knee areas were 
different. 


3.3 Pressure Measurements at Risk Area of Left Side-Tilt Position 


The pressure measurements in the side-tilt position to the left in high-risk areas are 
also presented in Table 2. All four pressure measurements of the shoulder and knee; 
elbow and hip; hip and knee, ankle were different. The pressure measurements of the 
ear and hip were different, except for maximal pressure. The pressure measurements 
of the ear and shoulder, elbow, knee, ankle; shoulder and elbow; knee and ankle were 
different, except for minimal pressure. Pressure measurements of the shoulder and hip 
were different, except for the area. The maximal pressure and area of the shoulder and 
ankle were different. The elbow and knee areas were different. There was no statistically 
difference between the elbow and the ankle. 


Table 2. At-risk area of pressure ulcer in side-tilt position. 


Parameters Average P Maximal P Minimal P Area (cm?) 
(N/cm?) (N/cm?) (N/cm?) 
Ear Rt. 0.43 0.53 0.32 40.32 
(0.40-0.46) (0.50-0.58) (0.25-0.37) (40.32—40.32) 
Lt. 0.38 0.49 0.28 40.32 
(0.34-0.40) (0.46-0.54) (0.18-0.32) (40.32-90.73) 
Shoulder Rt. 0.35 0.42 0.26 90.73 
(0.32-0.40) (0.38-0.46) (0.22-0.30) (90.73-90.73) 
Lt. 0.34 0.40 0.26 90.73 
(0.29-0.36) (0.36-0.43) (0.21-0.29) (90.73-90.73) 
Elbow (outer Rt. 0.24 0.29 0.21 20.16 
side) (0.21-0.36) (0.20-0.40) (0.17-0.30) (20.16-20.16) 
Lt. 0.27 0.29 0.23 20.16 
(018-—0.32) (0.19-0.35) (0.16-0.30) (20.16-20.16) 
Hip Rt. 0.45 0.51 0.39 90.73 
(0.43-0.50) (0.47-0.53) (0.37-0.45) (90.73-90.73) 
Lt. 0.44 0.49 0.38 90.73 
(0.42-0.46) (0.48-0.52) (0.35-0.41) (90.73-90.73) 
Knee Rt. 0.25 0.28 0.23 40.31 
(outer side) (0.22-0.27) (0.24-0.29) (0.19-0.26) (20.16-40.31) 
Lt. 0.23 0.27 0.20 40.32 
(0.21-0.28) (0.24-0.29) (0.18-0.26) (20.16-40.32) 
Lat. Ankle Rt. 0.21 0.25 0.17 20.16 
(outer side) (0.17-0.29) (0.19-0.36) (0.15-0.21) (20.16-20.16) 


(continued) 


226 J. Lee and J. S. Yoon 


Table 2. (continued) 


Parameters Average P Maximal P Minimal P Area (cm?) 
(N/cm?) (N/cm?) (N/cm?) 


Lt. 0.28 0.34 0.22 20.16 
(0.24-0.33) | (0.27-0.40) | (0.17-0.27) (12.60-20.16) 


Abbreviations: P, Pressure; Rt., Right; Lt., Left; Lat., Lateral 


3.4 Pressure Change Depending on Position 


To detect the pressure change depending on positions, we compared the areas with 
anatomical rotation. We matched the followings: head (occiput) on supine position and 
bilateral ears on side-tilt position, bilateral shoulder on supine position and bilateral 
shoulders on side-tilt position, sacrum on supine position and bilateral hip on side-tilt 
position, bilateral heel on supine position and bilateral lateral ankle on side-tilt position. 

Compared to the pressure on head (occiput) in supine position, average pressure was 
stationary on right ear and decreased on left ear. Also, maximal and minimal pressure 
were decreased on right ear and increased on left ear, respectively. In comparison of 
the pressure on bilateral shoulder in supine position and side-tilt position, average and 
maximal pressure were increased on right side-tilt shoulder, and minimal pressure was 
decreased on right side-tilt shoulder. In addition, maximal pressure was increased on left 
side-tilt shoulder; average pressure and minimal pressure were decreased on left side-tilt 
shoulder. Compared to the pressure on sacrum in supine position, all pressure parameters 
were increased on right hip. Also, average and minimal pressure were increased and 
maximal pressure was decreased on left hip. In comparison between the pressure on 
bilateral heal in supine position and bilateral lateral ankle in side-tilt position, all pressure 
parameters were decreased on bilateral side-tilt positions. 


4 Discussion 


Position change for reducing pressure and preventing ulcers have been the standard 
methods of patient care [1, 12, 15, 16]. However, studies showed failure in reducing the 
incidence of pressure ulcers [15, 18-20]. Although maintaining the skin contact pressure 
below 32 mmHg is expected to decrease the possibility of developing pressure ulcers, 
the effectiveness of repositioning patients for pressure sore prevention remains unclear 
[6]. 

Previous studies conducted with non-disabled population and pressure mapping sys- 
tem [5, 6]. They emphasized the high-risk of pressure area as peri-sacral area and greater 
trochanters. Our research was the first study which quantitively tracked the pressure 
parameters with position change and describe the clear pressure difference between 
each high-risk pressure ulcer areas. Furthermore, we clarified the importance of frequent 
position change in patient care for preventing pressure ulcers. 

Depending on the position, various findings were observed with quantitative mea- 
surements. In the supine position, the average, maximal, and minimal pressure measure- 
ments were higher on the head, sacrum, and coccyx than on the shoulder and heel. This 
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might be due to the bilateral distribution of body weight in the shoulder and heel. In the 
right and left side-tilt positions, we compared the right side and the left side of the body, 
respectively. The average, maximal, and minimal pressure measurements were higher 
on the bilateral ear than on the bilateral shoulder and elbow on the upper side of the 
body. In addition, these three pressure parameters appeared to be higher on the bilateral 
hip than on the bilateral knee and lateral ankle on the lower side of the body. 

Although there were minor differences in the composition of pressure measurements 
with statistical significance, we clarified the clear differences between pressure values 
in each risk area. In the supine position, most pressure parameters of the head, shoulder, 
sacrum, coccyx, and both heels showed significant differences, except between the right 
and left heels. In addition, in the right side-tilt position, all pressure measurements of 
the ear, shoulder, elbow, hip, knee, and lateral ankle were significantly different. In the 
left side-tilt position, most of the pressure parameters of the ear, shoulder, elbow, hip, 
knee, and lateral ankle were significantly different, except between the elbow and ankle. 
The difference between right-side tilt and left-side tilt might be explained by the small 
amount of recruited data, and it would be expected to be covered by larger scale analysis. 

In comparing descriptive measurements with positional rotation, the changes in pres- 
sure parameters at each high-risk area did not show a consistent pattern. This result 
suggest that it is more important to change the posture frequently than to make an ideal 
posture in patient care for preventing pressure ulcers. 

This study has some limitations. First, the pressure measurement by the sensor did 
not directly express the exact pressure loading on the tissue [20]. However, pressure 
sensor tracking remains one of the most popular methods for pressure monitoring. The 
clear presentation of pressure change in real time reflects a good representation of the 
pressure on the surface of the skin. Therefore, it allows us to intuitively recognize the 
increasing and decreasing pressures. Second, we only used the measured records of 
participants with a single brand of modern ward bed and sensor monitoring interface. 
However, we expected that the results would be similar, regardless of the type of hardware 
system. The pressure measurements at various positions are expected to be different for 
different beds/sensor mats; however, there is no doubt that the general trends should 
be reflected in the results. Finally, we studied the data of healthy adults, but not of 
patients. In patients with a high risk of developing pressure ulcers, the condition of the 
soft tissue in the affected area is clearly different from that of the general population. 
Therefore, there are many variables to consider regarding the risk factors for pressure 
ulcers, such as nutritional status, capillary blood flow, friction force and shear force. 
In this point of view, we considered that healthy participants were more likely to show 
a clear tendency of pressure measurements in each high-risk area at different positions 
and positional change. Further study of patients with classified impairments is necessary, 
because these results with normal population may appear differently depending on the 
patient’s variables. 


5 Conclusion 


This study quantitatively examined pressure distribution depending on position and 
found that the risk areas for pressure ulcers in different positions have different pressure 
distributions in a healthy population. These findings suggest clinical implications for 


228 J. Lee and J. S. Yoon 


positioning depending on patient impairment. Further studies with a larger patient popu- 
lation are needed to evaluate the actual risk of developing pressure ulcers in patients and 
to establish positioning methods to achieve optimum status to decrease pressure ulcer 
risk. 
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Abstract. Chronic Diseases are a prevalent problem that affects millions of people 
worldwide. It is a prevalent health condition that requires careful diet and medica- 
tion management and preventing chronic diseases. Traditional approaches to nutri- 
tional recommendation generation often rely on generic guidelines and population- 
based data, which may not account for individual dietary needs and preferences 
variations. In this paper, we propose a knowledge graph driven methodology 
for generating highly personalized nutritional recommendations that leverage the 
power of knowledge graphs to integrate and analyze complex data about an individ- 
ual’s health, lifestyle, and dietary habits. Our methodology employs a multi-step 
process that includes data collection and curation, knowledge graph construction, 
and personalized recommendation generation. We illustrate the effectiveness of 
our approach through a case study in which we generate personalized nutritional 
recommendations for a sample individual based on their specific health and dietary 
goals. 


Keywords: Chronic Kidney Disease - Knowledge Driven Approach - 
Personalized Nutritional Recommendation System 


1 Introduction 


Chronic kidney disease (CKD) is a long-lasting condition that can lead to a range of 
complications, including cardiovascular disease, decreased immune response, anemia, 
fluid retention and bone disease. Chronic kidney disease accounts for a very high risk for 
cardiovascular diseases [1] among other diseases. The high probability of developing 
conditions like hypertension can be countered to some extent using lifestyle changes 
such as increased physical activity, better dietary habits, and regular monitoring of med- 
ical reports related to these conditions. On the other hand, medical conditions such as 
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hypertension and diabetes can be successfully managed by changes in lifestyle such as 
dietary restrictions. These lifestyle changes include recommendations such as reduced 
salt intake as well as an increase in the consumption of fruits and vegetables while 
decreasing the consumption of processed food items [3]. 


1.1 Effect of Diet on CKD Patients 


Chronic kidney disease (CKD) is a condition in which the kidneys are not working as 
effectively as they should to filter waste products from the blood. Diet plays a critical 
role in managing CKD. Some of the effects of diet on CKD patients which we have 
taken Center of Disease Control and prevention [4] include sodium, protein, potassium, 
phosphorus, vitamins (such as vitamin D [5], iron [6], and calcium [7]), minerals, and 
fluids. 


1.2 Knowledge Graph-Based Application for Nutritional Recommendations 


In this paper, we propose a knowledge graph-driven methodology for generating highly 
personalized nutritional recommendations that leverage the power of knowledge graphs 
to integrate and analyze complex data about an individual’s health, lifestyle, and dietary 
habits. 

The development of applications that provide personalized health management and 
nutritional guidance [8] has the potential to improve the quality of life for CKD patients 
by facilitating self-management and providing access to specialized knowledge. In this 
paper, we present the development of an application that incorporates a range of features 
to support personalized health management for CKD patients. 

The app also includes medication reminders, nutritional guidance, drug interactions, 
CKD stage Calculation, and a log of medical tests and Food. This research paper aims 
to evaluate the effectiveness of our healthcare app in managing CKD. 

The application incorporates the following features: 


. Recommendation of nutrients depending on medical tests, 
. Log of medical tests, 

. Drug and Food interactions, 

. CKD stage calculation, 

. Noninvasive health data 


ABWN Re 


2 Related Work 


Previously work has been done regarding the implementation of tech-based solutions for 
chronic kidney disease most notably of which is the work done by Chatterjee et. Al [9] 
in his work they have taken a similar approach where lifestyle data can be logged into 
an app and because of it, a qualitative analysis of a patient’s lifestyle can be performed. 
Although the methodology proposed by this paper includes lifestyle data, it builds upon 
that by adding dietary data as well as medical test data performed by hospitals as well. 
Due to the addition of these parameters, the purpose of this paper is not only to help 
patients better understand the effect of their lifestyle choices but also as an analysis tool 
for medical professionals. 
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Another study that targeted the same domain created a web-based platform for help- 
ing women living in rural areas with obesity [10]. They used a web-based system to 
provide guidance tailored specifically to people living in hard-to-reach places. One of 
the limitations of that paper was that only participants that were interested in physical 
activity and nutrition were only interested in using the system. We have tried to solve 
the problem by taking proper input from the doctors as well and trying to incorporate 
the system into the treatment regimen of the patients. 

Another similar study [11] done related to this field provided a similar system to 
ours regarding the fact that the platform proposed by them had the ability to keep track 
of medical reports, physical activity, and dietary data and the data could be shared with 
medical professionals, but the platform was only targeted to meet the needs of pregnant 
women having diabetes. 


3 Methodology 


3.1 Overview 


Using an algorithmic approach, we developed a tool that generates personalized lifestyle 
recommendations for CKD patients based on their individual test results. The web appli- 
cation was developed using a combination of open-source tools and programming lan- 
guages, the architecture includes NodeJS, Express, MongoDB, React, and Tailwind CSS, 
D3 library and Knowledge graphs as shown in Fig. 1. The user data has been stored in 
MongoDB while the knowledge base is used to represent the interactions between the 
drugs, food etc. successfully divide the application specific data and the domain specific 
data. Along with this, patients will be able to log their non-invasive health data such as 
blood pressure, heart rate, and blood sugar levels. This would allow patients to monitor 
their health condition using a graphical user interface. The patients would also be able 
to log in to their medications to get drug interaction information about the medicine that 
they are taking. 


3.2 Server Side 


This system’s aim would be to ultimately provide recommendations regarding their 
health data and allow them a safe mechanism for self-monitoring that can also help the 
doctor regarding their medical decisions. We designed the prototype of this system using 
the agile methodology and as per the recommendations provided by it, we divided the 
development of the prototype into five parts: requirement gathering, requirement design, 
development iterations, testing, and finally feedback. 


3.3 Data Storage 


First, we collect data from various sources, such as medical records, wearable devices, 
and food intake logs, to create a profile of the individual’s health, lifestyle, and dietary 
habits. Finally, we generate personalized nutritional recommendations for the individual 
based on their specific health and dietary goals, using a combination of rule-based and 
data-driven approaches. 
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Fig. 1. Application Architecture 


Fig. 2. Ontology Design Methodology 


We have constructed a knowledge graph that represents the relationships between 
different entities in the data, such as foods, nutrients, health conditions, and lifestyle 
factors. To design the ontology, we used OD 101 as the knowledge base would be domain 
specific as shown in Fig. 2. We have used an RDF-based knowledge base that uses RDF, 
RDFS, and OWL to represent concepts in an interconnected form. Using an RDF triple 
store as shown in Fig. 3 allows us to publish the data in 4-star format according to Tim 
Berner Lee’s linked data criteria. This would also allow us to use the reasoning engines 
like the HermiT or Pellet reasoner which allows us to make decisions based on knowledge 
instead of data which helps us explain the reasoning behind the recommendations that 
are made. These reasoning engines also provide the advantage of finding associations in 
data that are not perceivable by the layman. 

Knowledge bases also provide a better and more natural format for drug interactions. 
In our application, this is best seen by the fact that drug interactions naturally have a 
symmetric property as shown in Fig. 4. That knowledge bases can very easily present 
while other data storage systems fail to do so. Domain and range restrictions also provide 
acheck for drug interaction data. The symmetric and reflexive axioms also provide a more 
natural representation of the knowledge represented in the knowledge base as all entities 
in a drug or food interaction inherently have a symmetric and reflexive relationship. 

The methodology followed by the recommendation system follows the following 
steps: 


1. The patients would log their health data including medical test reports, prescribed 
medications, etc. into the system which would be stored in a NoSQL database. 

2. Using the health data of the patient, the application would determine the medical test 
report values that fall out of the normal ranges. 
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Fig. 3. RDF Architecture 


3. Using this information, the application would query the knowledge base to find food 
items recommended for the patient given their medical test reports as shown in Fig. 5. 
This step is made possible mainly by the following object property: 


e interacts_with: this object property handles the interactions that take place between 
drugs and food items as well as drugs with other drugs. This object property is 
symmetric and hence has the advantage that the inference engine would complete 
any missing links in the knowledge base. 


Showing the dependencies between 10 classes 
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Fig. 4. Drugs/Food Interaction 


3.4 Domain Specific Features 


The most used method for calculating the stage of CKD is based on the estimated 
glomerular filtration rate (eGFR), which is calculated using the serum creatinine level, 
age, gender, and race of the patient. The eGFR provides an estimate of the rate at which 
the kidneys are filtering waste from the blood. The stages of CKD are classified based 
on the eGFR and the presence of kidney damage. 


PNRG: Knowledge Graph-Driven Methodology 235 


ones 
mente 
© © 2,0 
myin koo] tf oitor 
Greb SS JE a veces 
Cola L # rae 
=~ “Norvasc 
” own , iam 


Carp J : a 
= a d 


Covance 
Sars 
etree 


Fig. 5. Relationship between drugs and foods 


4 Evaluation Methodology 


We had 10 volunteers of different genders, heights and weights and age groups, and 
through CKD stage calculation we have grouped them into 5 groups as shown in Table 1. 


Table 1. Details of Volunteers 


Subject Age Gender Height (cm) Weight (kg) CKD Stage 
S1 24 Male 182 74 3B 
S2 29 Female 162 59 

S3 61 Male 177 95 4 
S4 37 Male 179 84 3A 
S5 57 Female 160 82 2 
S6 49 Male 170 70 3A 
S7 55 Female 158 70 4 
S8 72 Male 174 80 3B 
S9 42 Male 176 70 1 
$10 66 Female 163 72 2 


We recommended nutrients based on their medical tests and recorded their noninva- 
sive health data e.g., blood pressure and heartbeat for 6 weeks. Over this specific period, 
with proper recommendations and logging of their health data and medical records, they 
were able to control their disease in a more efficient way, as the feedback taken from the 
survey, we conducted from the volunteers as shown in Table 2. 
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Table 2. Survey conducted of volunteers for reusability of application. 


Points Min | Max | Average 
User Friendliness 7 10 8.6 
Effect on general lifestyle 7 10 8.3 


How likely to keep using the | 8 10 8.4 
app 
Recommend to a friend 8 10 8.3 


5 Result 


The development of a prototype done through the agile methodology allowed us to 
regularly obtain feedback from patients as well as health professionals. The product 
contained proper visualizations of both types of data: numerical and categorical. This 
allows the patients as well as health professionals to quantitively assess the lifestyle of a 
patient during a normal day. The prototype can also provide the possibility for patients 
to set goals for their day which includes exercise time etc. 


6 Conclusion 


By leveraging the power of knowledge graphs to integrate and analyze complex data, 
our approach can generate highly personalized recommendations that are both effective 
and sustainable. Our research findings suggest that personalized recommendations can 
significantly improve patient outcomes and reduce the progression of CKD. By using 
the app, patients can track their progress, receive guidance, and support, and connect 
with their healthcare provider. We believe that our healthcare app has the potential 
to revolutionize CKD management by providing patients with a comprehensive and 
personalized approach to their care. We encourage CKD patients to download and use 
the app to take control of their health and manage their condition effectively. With the 
right tools and support, CKD patients can lead fulfilling and healthy lives. Personalized 
lifestyle recommendations based on individual test results can improve the outcomes of 
CKD patients. 

Further research is needed to evaluate the effectiveness of this approach in a clinical 
setting. 


7 Future Work 


RDF being a non-proprietary format, also has the advantage that by providing access 
through the SPARQL endpoint, the knowledge base can potentially act as an open-source 
repository. This would allow a centralized repository containing all drug data as well as 
data related to their interactions with other drugs, food items and medical conditions. 
Although our current ontology has been designed to only accommodate use cases for 
drug-drug and drug-food interactions, the ontology can be expanded to accommodate 
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interactions between drugs and medical conditions as well as drug and genetic variations. 
This presents the opportunity to increase the scope of the application in the future 
from assisting doctors with providing them with insights about the patient’s lifestyle 
and providing recommendations related to the food and drug recommendations, to the 
system providing personalized recommendations based on the patient’s invasive as well 
as non-invasive medical data such as genetic data. An example of a drug interaction with 
genetic properties individuals is where the optimal dosage of Warfarin, an anticoagulant 
that acts by reducing the activity of vitamin K, is dependent on the patients CYP2C9 and 
VKORCI genotype [12]. The same principle applies with medical conditions affected 
by drugs. Acetaminophen, commonly known as Tylenol if taken in high doses can have 
adverse reactions in kidney disease patients [2]. 
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Abstract. It is difficult for inpatient rehabilitation patients to continue to per- 
form rehabilitation exercises in the community after leaving the hospital. This is 
because various exercise programs, which are not medically proven, do not reflect 
the specificity of the individual and are performed collectively due to adminis- 
trative and financial convenience. The purpose of this study is to evaluate and 
compare the effects of exercise programs using the Smart Elephant total body 
exercise device and walking on mental and physical outcomes with real-time 
monitoring to develop a customized rehabilitation exercise program optimized 
for people with disabilities. To conduct this study, five non-disabled people liv- 
ing in the community were selected to participate in the exercise programs of 
Intervention A (walking), Intervention B (walking and cycling), and Intervention 
C (cycling) for 9 weeks to determine the effects on physical function measures, 
psychosocial, mental and quality of life health outcomes, participants’ feedback 
and satisfaction surveys, and changes in Electromyography (EMG), Electroder- 
mal Activity (EDA), Temperature (TEMP), and oxygen saturation (SpO2) during 
the intervention. It is believed that it can be used as a basis for customized reha- 
bilitation exercise that provides a validated rehabilitation exercise service model 
for people with disabilities in the community. 


Keywords: Whole Body Exercise Machine - person with Disability health - 
exercise program 


1 Introduction 


As living standards improve, access to healthcare expands, and people become more 
concerned about their health, morbidity, mortality, and life expectancy are increasing 
(Jang & Ham, 2014). However, unlike the development of medical technology, it is diffi- 
cult for rehabilitation patients who were hospitalized to continue rehabilitation exercises 
in the community after leaving the hospital. 
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Rehabilitation has been shown to improve mobility, physical functioning, and quality 
of life for people with disabilities suffering from illness. Mental health improvements 
(Blondell et al., 2014). However, while there are many companies developing equipment 
for older adults and people with disabilities, many of the products developed are not 
clinically validated, so their use is problematic and few are actually used by older adults 
and people with disabilities. The Elephant Total Body Exerciser is a total body exercise 
device that is consistently sold to neurological and musculoskeletal patients. It is used 
as an upper and lower limb rehabilitation device for people with disabilities in most 
rehabilitation hospitals and nursing homes. In addition, a study (Shin, 2016) analyzing 
the design satisfaction of rehabilitation medical devices using an ergonomics perspective 
(Shin, 2016). 

The purpose of this study is to provide a validated rehabilitation exercise service 
model for people with disabilities in the community through a customized rehabilitation 
exercise study optimized for individuals by evaluating and comparing the effects of smart 
elephant exercise devices and walking exercise programs with real-time monitoring on 
mental and physical outcomes. To this end, we will confirm the effects of exercise 
programs on assessment tools and vital signs through a study on non-disabled people 
before conducting a study on disabled people. 


2 Method 


2.1 Participants 


To conduct this study, we used five community-dwelling, non-disabled individuals 
(Table 1). According to Virzi (1992) and Nielson (2000), five users are the most efficient 
way to identify usability and problems of a particular product, and they reported that 
80% of the problems can be found arithmetically. The inclusion criteria for the study 
were adults aged 18 years or older, at least 3 months since the diagnosis of any type of 
disease, free of cognitive and psychological disorders, able to communicate, and able 
to understand the purpose of the study and participate in the study. Exclusion Criteria 
is Subjects were excluded from the study if they had neurological or musculoskeletal 
disorders that made it difficult for them to use a whole body exercise device, if they were 
participating in other organized research projects, or if they had medical problems that 
made it difficult for them to participate in the study, as determined by the healthcare 
provider. 


2.2 Smart Elephant Whole Body Exercise Machine 


The upper and lower extremity rehabilitation exercise machine is a combined upper and 
lower extremity rehabilitation device that focuses on basic and essential rehabilitation 
treatments such as strengthening, preventing contractures, and increasing joint range of 
motion. It consists of an aluminum wheel that turns a weighted wheel that is driven evenly 
by a chain connected to a rotating body as the crank rotates, a brake device that stops 
the weighted wheel, and a load adjustment device using magnets, so it is a device. By 
applying healthcare monitoring technology to this device, it provides feedback to each 
user on the number of times, time, resistance, and intensity to enhance the effectiveness 
of exercise. 
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Table 1. General characteristics of participants (n = 5) 


Case No Gender Age (years) Relevant Health Information 
1 F 23 No specifics 
2 23 No specifics 
3 F 22 No specifics 
4 F 22 No specifics 
5 F 21 No specifics 


2.3 Study Protocol 


This study used a Single Experimental Research Design, the ABC Design, to clarify 
the cause and effect of the intervention in five non-disabled individuals. The exercise 
program consisted of Intervention A (walking), Intervention B (walking and exercise 
machine), and Intervention C (exercise machine) for 9 weeks, with each intervention 
lasting 3 weeks. Each intervention was conducted twice a week for 30 min, for a total 
of 6 sessions, and to determine the effectiveness of the exercise program, biosignal data 
was measured in real time and monitored by the user through a biosignal meter called 
Biosignalplux Professional. An intervention evaluation tool consisting of a physical test, 
fall risk assessment, psychosocial test, mental health test, and satisfaction test was con- 
ducted as a pre-assessment before intervention A, a post-assessment after intervention 
A, a post-assessment after intervention B, and a post-assessment after intervention C. 
The intervention evaluation tool was administered to the participants. 


2.4 Evaluation 


In this study, several papers were reviewed to evaluate the effect of exercise programs on 
the physical and psychosocial, and among them, representative evaluation tools used for 
the disabled in the physical and mental domains, which are necessary factors to improve 
the quality of life, were used as follows. 


2.4.1 Physical Function Measures 


To measure physical function measures, we used the Borg Rating of Perceived Effort 
(RPE). This tool is roughly equivalent to heart rate per minute as an indicator of exercise 
intensity (Kwon et al., 2018). 


2.4.2 Psychosocial, Mental and Quality of Life Health Outcomes 


To measure psychosocial, mental and quality of life health care outcomes, we used the 
Korean version of the WHOQOL-BREF, which was developed by the World Health 
Organization to measure quality of life for use in all cultures and standardized by Min 
(2000). 
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2.4.3 Participants’ Feedback and Satisfaction Surveys 


To evaluate participants’ feedback and satisfaction surveys, we used the Self Efficacy for 
exercise (SEE) instrument, which assesses self-efficacy for exercise and is self-reported. 
Self-efficacy has been cited as an important factor influencing physical activity (Lee 
et al., 2007). 


2.4.4 Biosignal Acquisition Applications 


Measure biosignals in real time while performing a task. The tool used is the Biosig- 
nalplux Professional. It can provide up to 10 h of continuous signal streaming wirelessly 
at 16-bit resolution per channel at a sampling rate of up to 3.000 Hz with eight special- 
ized sensors of your choice, and it can record eight data simultaneously. The vital signs 
to be measured by this tool. 


2.4.4.1 Electromyography (EMG). Measuring muscle activation by bioelectricity. In 
this study, surface electromyography was used to quantitatively measure maximal vol- 
untary contraction values. The sampling rate was 1000 Hz and the notch filter was 60 Hz. 
The measurement site was selected for the externsor muscles, which are the most used 
muscles when using the whole body exercise device, and the sensor Ch.1 was measured 
for the left biceps brachi, Ch.2 for the right biceps brachi, Ch.3 for the left quadriceps, 
and Ch.4 for the right quadriceps. 


2.4.4.2 Electrodermal Activity (EDA). By measuring the change in sweat secretion, 
the sweat glands are active and the sympathetic nervous system is active. In this study, 
sensors were attached to the 2nd and 3rd fingers, and changes in the average value of 
skin conductance were measured in the signal parameters of each intervention. 


2.4.4.3 Temperature (TEMP). It is used to measure physical or ambient temperature 
and provides an analog output of temperature change, with a short response time and high 
accuracy. In this study, the temperature was measured immediately after the intervention 
to check the body temperature and to measure the change between each intervention. 


2.4.4.4 Oxygen Saturation (SpO2). The SpO2 sensor is designed to estimate the oxy- 
gen saturation level in the finger using two LEDs, and the reflected light from each of 
these LEDs is absorbed by a photodiode that converts the current into a digital value, 
from which the blood oxygen saturation can be derived. In this study, the average oxygen 
saturation value was measured to determine changes in the intervention. 


3 Result 


3.1 Physical Function Measures 


The results of the paried t-test for within-group comparison of PRE at pre test, post test 
1, post test 2, post test 3, and post test 4 are shown in Table 2. In RPE, the score was 
7.2 + 0.44 at pre test and 9.6 + 1.14 at post test 1, which was a statistically significant 
improvement (P < .05). In post test 2, the score was 11.6 + 1.14, a statistically significant 
improvement compared to post test 1 (P < .05). In post test 3, the score was 14.4 + 2.07, 
a Statistically significant improvement compared to post test 2 (P < .05). There was also 
a Statistically significant improvement when comparing pre test to post test 3 (P < .05). 
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Table 2. Characteristic of the RPE Test of Total Score 


RPE | Mean + SD P-value 
pre Post 2 Post 3 Pre Post 1 | Post 2 | Pre 
-post 1 | - Post |- Post | -post 3 
2 3 


Total 7.2 +0.44 9.6 1.14 | 11.6 + 1.14 | 14.4 + 2.07 | 0.009" | 0.011* | 0.009" | 0.001" 
* P< .05 


3.2 Psychosocial, Mental and Quality of Life Health Outcomes 


The results of the Wilcoxon signed rank test for within-group comparisons of WHOQOL- 
BREF at pretest, post test 1, post test 2, and post test 3 are shown in Table 3. The 
WHOQOL-BREF showed a statistically significant improvement from 79.8 + 8.31 at 
pretest to 81.8 + 8.58 at posttest 1 (P < .05). In post test 2, the score was 81.8 + 10.47, 
which was not statistically significant compared to post test 1 (P > .05). In post test 3, the 
score was 85.0 + 9.82, which was not a statistically significant difference compared to 
post test 2 (P > .05). There was a statistically significant improvement when comparing 
pre test to post test 3 (P < .05). 


Table 3. Characteristic of the WHOQOL-BREF Test of Total Score 


WHOQOL-BREF | Mean + SD P-value 
pre Post 1 Post 2 Post 3 Pre Post1 Post2 | Pre 
-post 1 | -Post2 | -Post3 | -post3 
Total 79.8+8.31 |81.8 +8.58 | 81.8+10.47 | 85.0+9.82 | 0.042" | 1.000 | 0.273 | 0.043" 
*: P < 05 


3.3 Participants’ Feedback and Satisfaction Surveys 


The results of Wilcoxon signed rank test for within-group comparison of SEE in pre 
test, post test 1, post test 2, and post test 3 are shown in Table 4. SEE scored 782.0 + 
213.23 in pre test and 878.0 + 182.67 in post test 1, which is not a statistically significant 
difference (P > .05). In post test 2, the score was 878.0 + 182.67, a statistically significant 
improvement compared to post test 1 (P < .05). There was no significant difference in 
post test 3, with a score of 880.0 + 275.77, compared to post test 2 (P > .05). There 
was also no statistically significant difference when comparing pre test to post test 3 (P 
> .05). 


3.4 Electromyography (EMG) 


The results of the Wilcoxon signed rank test for within-group comparison of EMG in 
interventions A, B, and C are shown in Table 5. In EMG, it was 0.74 + 0.43 in A and 
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Table 4. Characteristic of the SEE Test of Total Score 


SEE | Mean + SD P-value 


pre Post 1 Post 2 Post 3 Pre Post1 Post2 | Pre 
-postl | -Post2 | -Post3 | -post3 


Total | 782.0 + 213.23 | 878.0 + 182.67 | 878.0+ 182.67 | 880.0+ 275.77 | 0.686 | 0.041" | 0.498 0.225 
* P < .05 


0.72 + 0.35 in B, a statistically significant improvement (P < .05). In C, there was no 
statistically significant difference compared to B at 0.80 + 0.31 (P > .05). There was 
also no statistically significant difference when comparing A and C (P > .05). 


Table 5. Characteristic of the EMG Test of Total Score 


EMG Mean + SD P-value 
A B C A-B B-C A-C 


Total 0.74 + 0.43 0.72 + 0.35 0.80 + 0.31 0.552 0.035" 0.086 


3.5 Electrodermal Activity (EDA) 


The results of the Wilcoxon signed rank test for within-group comparison of EDA in 
interventions A, B, and C are shown in Table 6. EDA was 4.22 + 1.82 in A and 4.27 + 
2.23 in B, which was not statistically significant (P > .05). In C, the score was 5.76 + 
1.33, a statistically significant improvement compared to B (P < .05). When comparing 
A and C, there was a statistically significant improvement (P < .05). 


Table 6. Characteristic of the Electrodermal Activity of Total Score 


EDA Mean + SD P-value 
A B C A-B B-C A-C 
Total 4.22 + 1.82 4.21 + 2.23 5.76 + 1.33 0.992 0.004” 0.001* 
* 

:P < .05 


3.6 Temperature (TEMP) 


The results of Wilcoxon signed rank test for within-group comparison of Temporature 
in A, B, and C interventions are shown in Table 7. Temporature was 36.53 + 0.22 in 
A and 36.55 + 0.11 in B, which was not statistically significant (P > .05). In C, it was 
36.57 + 0.09, which is not a statistically significant difference compared to B (P > .05). 
There was also no statistically significant difference when comparing A and C (P > .05). 
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Table 7. Characteristic of the Temperature of Total Score 


TEM Mean + SD P-value 
A B C A-B B-C A-C 
Total 36.53 + 0.22 36.55 + 0.11 36.57 + 0.09 0.768 0.401 0.342 
* 

:P <.05 


3.7 Oxygen Saturation (SpO2) 


The results of Wilcoxon signed rank test for intra-group comparison of oxygen saturation 
in A, B, and C interventions are shown in Table 8. The oxygen saturation was 98.50 + 
0.68 in A and 98.56 + 0.50 in B, which was not statistically significant (P > .05). In 
C, it was 98.50 + 0.50, which was not statistically significant compared to B (P > .05). 
There was also no statistically significant difference when comparing A and C (P > .05). 


Table 8. Characteristic of the oxygen saturation of Total Score 


SpO2 Mean + SD P-value 
A B C A-B B-C A-C 
Total 98.50 + 0.68 98.56 + 0.50 98.50 + 0.50 0.796 


“P< 05 


4 Discussion 


The purpose of this study was to evaluate and compare the effects of real-time monitor- 
ing smart elephant whole body exercise device and walking exercise program on mental 
and physical outcomes, and to study customized rehabilitation exercise optimized for 
people with disabilities. To this end, we evaluated and compared the effects of using the 
smart elephant whole body exercise device and walking exercise programs on mental and 
physical outcomes in five non-disabled people in the community. The results showed 
that the physical function measures, PRE assessment, showed a significant improve- 
ment between each intervention from pre test to post test 3, and the WHOQOL-BREF of 
psychosocial, mental and quality of life health outcomes showed that walking exercise 
affected the psychosocial, mental and quality of life health outcomes more than before 
exercise, and the elephant whole body exercise device affected the psychosocial, mental 
and quality of life health outcomes more than before exercise. In other words, the effects 
of exercise on psychosocial, mental and quality of life health outcomes could be seen, 
but there were no differences between each exercise program. In the SEE, which exam- 
ined participants’ feedback and satisfaction surveys, there was no difference between 
pre- and post-exercise, but the combined walking and elephant exercise program had a 
impact on participants than the walking program. In other words, The study results indi- 
cate promising improvements in physical function measures, psychosocial and mental 
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health outcomes, as well as participants’ feedback and satisfaction. The observed sta- 
tistically significant improvements in the Borg Rating of Perceived Effort (RPE) and 
the WHOQOL-BREF scores support the effectiveness of the exercise interventions. The 
findings suggest that the Smart Elephant Whole Body Exercise Machine and walking 
exercises have a positive impact on the physical and mental well-being of individuals 
with disabilities. 

The biosignal data showed that the walking plus elephant exercise program had a sig- 
nificant improvement in EMG compared to walking, and the elephant exercise program 
had a significant effect on EDA compared to walking and walking plus elephant exercise. 
However, there were no statistically significant differences between the interventions in 
temperature and oxygen saturation. 


5 Conclusion 


It was necessary to standardize the effects of the exercise program by conducting a study 
on non-disabled people before conducting a study on people with disabilities. Also, a 
study on non-disabled people was conducted before the mid- to long-term project for 
people with disabilities to identify deficiencies in safety and operation. A limitation of the 
study is that it is difficult to generalize the results because the experiment was conducted 
only with women in their early twenties. Therefore, in future studies, an appropriate 
number of sampling and balanced sex subjects are needed to increase statistical signif- 
icance. In addition, since it was conducted for non-disabled people, not the disabled, 
it was not possible to use the initially set evaluation tools. Evaluations in the areas of 
daily living and falls were excluded from the results because it is difficult to compare 
and evaluate the functions of people with disabilities because their functions are perfect 
and do not change. In addition, we tried to check the electrocardiography (ECG) of vital 
signs, but the sensor generated a lot of noise during exercise, so we could not extract 
accurate data, so it was excluded from the results. 

Due to the improvement of living standards, expanding access to medical services, 
and increasing interest in people’s health, disease-related mortality and life expectancy 
are gradually increasing, but unlike the development of medical technology, it is difficult 
for rehabilitation patients who were hospitalized to continue to perform rehabilitation 
exercises in the community after leaving the hospital. This is because various exer- 
cise programs that are not medically proven do not reflect individual specificity and 
are performed collectively due to administrative and financial convenience (Bae et al., 
2010). This study will be used as a basis for customized rehabilitation exercises that 
provide a proven rehabilitation exercise service model for people with disabilities in the 
community, and it is hoped that the lives of people with disabilities will become health. 
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Abstract. In the past decade, a lot of challenges to access, assess, and to acquire 
the needed technological opportunities to teach computers what naturally comes 
from the human brain and to understand how we naturally react when we rely on 
technology. The ability to document human thoughts, reactions and behavior to 
computers has led to the coming of NLP, AI, DI, & ML. Aim to understand the 
influence of IoT on humans with the use of DL to achieve content correctness and 
accuracy with virtual technology. Studies show that the way we think, react, and 
do the things we think “Internet of thoughts” reflect our personality. The way we 
think determines the way we react and the way we do things are based on how we 
think. Technology advancement has reinforced a lot of changes in humans which 
makes humans vulnerable to personal content exposure misappropriation due to 
the continuously changing nature of humanity and language. The study uses NLP, 
DL and behavior-oriented drive and influential function and results show that IoT 
based on VR influences human psychology “Internet of Thoughts”. 
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1 Introduction 


Deep learning (DL) is a computer program called machine learning that provides access 
to techniques that teach computers to do what comes naturally to humans [1]. Natural 
language processing (NLP) is a subfield of computer linguistics, computer science, and 
artificial intelligence (AI) that connects and enables the interactions between computers 
and human natural language. Technology advancement has reinforced a lot of changes 
in humans [2]. The precipitated changes in the internet world have modified the way we 
see the outside environment, manage our affairs and make changes to our bodies and 
environment. Due to precipitated changes in internet things (IoT), the attraction effect 
on human psychology “internet of thoughts” (IoThs) now occurs when the preference 
for a target increases relative to our choice option that is dominated by the target- 
oriented services of internet services. The eye is an important part of human life [4]. 
The way we see things determine the way we react to them in real life. Virtual reality 
(VR) helps to redesign prior reactions and feelings through virtual performance [5]. The 
eyes see content different from virtual technology [9]. Virtual reality build in humans 
a strong emotional engagement with data and can help tremendously in data extraction 
[7]. It shapes the way we see the world around us. Virtual reality (VR) is a computer- 
simulated experience that employs pose tracking and 3D near-eye displays to give users 
an immersive feeling of a virtual world via hardware (headsets) and software designers to 
create VR experiences such as virtual museums [8]. Due to these advances in technology, 
human react responses and behaviors to real-world natural artifacts for museums are 
now perceived differently because of virtual reality [6]. There exist a new struggle to 
balance the much important technology such as virtual reality and the physical facilities 
represented in 3D systems and object identifiers by virtual reality [10]. Natural language 
processing (NLP) is a sub-field of artificial intelligence (AI) that uses computers to 
understand, analyze, and extract meaning from human language in a smart, sustainable, 
and useful manner Developers uses natural language processing to organize structure 
knowledge to perform tasks such as automatic summarization, translation, named entity 
recognition, relationship extraction, sentiment analysis, speech recognition, and topic 
segmentation [3]. Natural language processing uses common word processor operations 
to handle text and treat them as a sequence of symbols taking appropriate consideration 
of the hierarchical structure of language. 


2 Literature Review 


The internet of thoughts (IoThs) “Internet of Thought (IoThs)” is part of the human 
mental process that built psychological associations with models of computer systems 
and uses the products and services of the internet of things to influence the way we 
think, react and do things based on how we think. The development of models and 
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modeling systems software that represents human natural thoughts has allowed other 
technologies like the internet of things to influence the way we think, react and do 
things the way we think. One of the advantages that IoTs have over humans is their 
ability to connect living things and non-living. The process of the internet of things is 
far beyond human capacity and it’s influencing the way humans think, act and behave. 
The psychoanalytic theory explains that cognitive psychology engages humans with 
things based on the way they see, access, and feel about them. The higher the cognitive 
processes the more deeply the influence on human ways of think, acting, and doing 
things. Identifying things and resolving differences with items of similar nature have 
a higher achievement with the internet of things smart grip. The deep desire humans 
experience when in an encounter to identify an item, exerts a tense atmospheric situation 
in humans. The inherent feelings that arose during a moment of identifying amongst 
alternatives relate to Freud’s thermodynamics theory acknowledgments power of internet 
service. Due to advanced developments, identifying someone based on his or her original 
and choice of words is becoming increasingly possible. This has made it easier for 
internet service to influence human choice, The advances in IoTs have a strong grip on 
human intuitive attention, decision making, learning, judgment, reasoning, thinking, and 
cognitive processes. 


2.1 Basic Seven (7) Principles of Unprecedented Changes on Human Psychology 
“Internet of Thoughts” 


The study provides an explanation to the unprecedented changes in the way we think, 
react and do things we do think in terms of seven (7) principles below. The following 
definition explains the role of these elements in changes in our activities. 


Cultural Heritage. This relates to the culture we inherited from birth from where we 
grow up. The language we speak depicts part of our culture. Certain languages are asso- 
ciated with certain cultures. In the world, the English language depicts British culture, 
American English depicts American culture, French culture depicts French culture and 
the Polish language depicts Polish culture. Cultural heritage offers a key opportunity for 
the development and deployment of new IoTs systems, with potential benefits both for 
the culture [11]. 


Educational Standards. The educational standard of a person changes the way he or 
she thinks, reacts, and does things. It is often very easier to judge people just by way 
of observing them act, speak and behave. The educational status of someone instills a 
certain culture in the person’s altitude. It is easier to be connected with real-world affairs 
by being acknowledged the all-around happenings [12, 22]. 


Environmental Situation. The environment in which people grow pre-determine some 
of their altitudes. Persons in urban areas have a way of behaving that is different from 
those in rural areas. It is understood that the urban environment exposes people more 
than the rural environment. The growth of skill enhancement expands one’s sense of self 
and adaptation to the environmental and personal growth facilitation model of curiosity 
informs our research [13]. 
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Information Availability. The availability of information changes the way we think, 
react and do things. The availability of information has a way to instill a degree of 
culture in us. Having a long list of alternatives normalizes internal desire. It is known 
that information is the power to mental growth. Having information helps to advance 
moderate choices as compared to the situation with no or limited access to information. 
Information availability allows one to suggest that information availability has an effect 
on the habit of thinking [14, 18]. 


Information Accessibility. The way we think, react, and do things depend on informa- 
tion accessibility. This has a great way to influence us. There are some countries with 
little or no free access to certain information which course citizens of such countries to 
have a different perception than others who have access. The availability of access is 
the availability of knowledge. The accessibility of cue information in memory affects 
which decision strategy individuals rely on [15]. 


Information Efficiency. The ease with which we obtain information helps a lot to shape 
our way of thinking, reacting, and doing things. Some peoples turn out to be very selfish 
with knowledge and information. A wide range of factors influencing the individual. 
Subjective norm is activated when individuals become aware that certain behaviour that 
they perform have adverse effects on issues they belief, and that behavioural action will 
have a positive and significant impact on the aspects the individual values [18]. 


Cognitive Resource Efficiency. There is a quality achieved such as an experience, 
intelligence, competence, and task-relevant knowledge [17]. The flexible allocation of 
cognition during translation is assumed to codetermine overall efficiency in translation 
[16]. 


2.2 The Theory of Psychoanalysis Applied Basic Seven (7) Principles 
of Unprecedented Changes on Human Thoughts 


The figure below explains the relationship between the identified elements known 
as principles of unprecedented changes on human thoughts with Freud’s Theory of 
thermodynamics of psychoanalysis. 

Figure | represents ID which is the Unconscious drive by IoTs. Pleasure-oriented 
and self-drive by inherent instincts by the internet which can be attributed to cultural 
heritage, educational standards, and environmental situation. EGO is the Conscious- 
ness rational driven by situational reality. The balance demand of the Id and superego 
can be attributed to information availability, information accessibility, and information 
efficiency. SUPEREGO is the state of Morality Concerned. The personality developed 
through socialization is attributed to cognitive resource efficiency. 
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SUPEREGO 


“ris not right time to 
get it now.” 


EGO 


The superego | Morality Concerned. The 
personality developed 


through socialization. 


Fig. 1. Psychoanalysis applied basic Seven (7) principles of unprecedented changes on human 
thoughts. Source: author’s own copy 


2.3 Deep Learning Interrelation with Natural Language Processing and Machine 
Learning 


The section represents information, applications, and system software that enable and 
enhances data extraction. Most data from simple text, to classifications, patterns and 
clustering pass through one of the following programming languages below depending 
on the context of information. 


Machine Learning 


GE = 


Deep Learning 


OUT PUT 
LAYER 
Deep Learning 


[ Future Erection HE Gassincation | 


Fig. 2. Deep learning interrelation. Source: author’s own copy 


Figure 2 represents components of natural language processing, deep learning, and 
machine learning. Over the past years, the field of natural language processing has 
been propelled forward by an explosion in the use of deep learning models [19]. As 
natural language processing and Machine Learning (ML) tools rise in popularity, it 
becomes increasingly vital to recognize the role they play in shaping societal biases [20]. 
Recently, the emergence of pre-trained models (PTMs) has brought natural language 
processing (NLP) to a new era. After the advances made in Computer Vision using deep 
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learning tools, NLP has metamorphosed recently. Deep learning is represents learning 
algorithms in deep learning that natural language processing uses to run its applications. 
Applying deep learning to solve fundamental problems in NLP has tremendous strides in 
deep learning applications to NLP. Recently, the NLP community has witnessed many 
breakthroughs due to the use of deep learning. Deep learning (DL), is a subfield of 
machine learning (ML) [21]. 


3 Results 


In this section, we provide information on how text or speech content can be extracted 
and classified into different parts of the speech based on data obtained via NLP means 
that help in decision-making for digital services. The influence is symbolized as BIF 
= F(D) which is said “f of d” equal to Eq = f py x BS Metrics range is up of 
(nouns, adjectives, verbs, adverbs, interjections, prepositions, conjunctions, pronouns, 
determiners, and numerals). Metrics range substitute is made up of (nouns, adjectives, 
verbs, adverbs, interjections, numerals, and prepositions). 


fo aN 
( Cas ) 
>~ 
y 
f Have, fever, nausea, \ EN bpe 
7 headache, vomits, / Al omen, pain, upset 7 
tis Diarrhea, Abdominal, T | peta stomach, appetite, lss, MR=11 
indigestion, blood 


Faienh | MEU a | 


Fig. 3. Metric range and metric range substitute. Source: author’s own copy 


vomits 


Above Fig. 3, provide data classified into different groups Class A and Class B. 
The group mark with red are made up of metric range substitute and metrics range 
for a single individual called Class Band that with green is called Class Awhich con- 
stitute of metrics range substitute and metrics range. To obtain influence rate, metric 
range substitute is divided by sum of metric range then multiple by behavior score. 
Class A MR =11 and MR*=9 while Class B :::)) MR =12 and MR*=11. 
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SOLUTION CLASS A SOLUTION CLASS B 
Class A MR =11 and MR*=9 Class B MR =12 and MR*=11 
MR=Merrics Range MR=Metrics Range 
MR*= Metrics Range Substitute MR*= Metrics Range Substitute 
BS=Behavior Score (Also human five BS=Behavior Score (Also human five 
sense organs) sense organs) 
i + 

Eq=f (D) diih BS Eq=f (D) #8) xBS 

==-)))) ===) 
f(D) = į x5 fw) = x5 

====))) f(D) = 4.0909 ====))) f(D) = 4.58 


From the statistics we can say that the influence score is grade “ Very Good” as per 
classification is achieve by both class but Class B is better than Class A with behavior 
score of 4.58 and 4.0909 respectively. 


4 Applied Method 


The section explains the stages used to develop the classification and how data were 
selected and classified into a deep learning model. The study uses two sample texts and 
is classified into groups called Class A and Class B as per Fig. 3. This section represents 
the various parts of speech selected by the author. We consider these parts of speech 
as those that convey information in a simplified manner to the lowest level that even a 
language beginner can learn from. 

Figure 4 represents a deep learning model of a recurrent neural network. The hidden 
layer is represented by ten (10) parts of speech. The ten (10) parts of speech are grouped 
into two fractions called metrics range substitute (nouns, adjectives, verbs, adverbs, 
interjections, numerals, and prepositions) and metrics range (nouns, adjectives, verbs, 
adverbs, interjections, prepositions, conjunctions, pronouns, determiners, and numerals). 
The metrics range substitute detailed summaries of a speech and provide meaning in the 
simplest form while the metrics range contains all of the parts of speech. Datasets are 
automatically differentiated in the hidden layer sector and sent to the output with two 
layers. 

Figure 5 represents the behavior layers of content extraction charge of generating 
coordinated verbal gestural and facial behaviors for realizing an important of each utter- 
ance behavior to the require context. This help the extractor or practitioner determine 
influence of each utterances. This includes the synthesis of intelligible verbal and gestu- 
ral acts per see and each combination allows a continuous flow of human-like multimodal 
utterance as digital services streamline. The main research purpose of this algorithm is 
to analyze the degree of richness user content. 
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I, I, and, 


Fig. 4. Applied Deep Learning on Content Extraction using Natural Language Processing. 
Source: author’s own copy 


Fig. 5. (Scoring model) Virtual reality and internet of things on human psychology. Source: 
author’s own copy 
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5 Virtual Reality and Internet of Things Influence Human 
Psychology 


This section presents the parts of virtual reality that engages with the internet of things 
to influence human’s way of thinking, reacting, and responding to situations. Humans 
are face with challenges of complex system, applications, and multiple technologies 
(Fig. 6). 


| ENGAGENSENT | "i 


IMAGINATIOM 


Fig. 6. Virtual reality and internet of things influence human psychology. Source: author’s own 
copy 


There is no technology that was aim to confuse uses except a redirection of the 
intended use for something locally. This has been the work of haters of healthy and 
secure world. This study examine four (4) aspects of virtual technology that internet 
of things have fully integrated to give humanity good life. These four parts of virtual 
reality technology engage the text and images of a user. Virtual reality technology is a 
cutting-edge intelligent system that is currently processing the connecting technology 
of artificial psychology and is also gradually coming to maturity with the help of the 
Internet of Things. 


Immersion and the Internet of Things on Human Psychology. Immersion is simply 
a sense of reality that promote interest and encourage participation. Immersion in the 
internet of things helps humans’ to boost their way of reasoning with a comprehensive 
technology that combined with high technology to achieve a multi-field study under the 
virtual background using the internet of things to connect with the outside world and to 
realize the interactive feelings. Without the internet of things, there will be no virtual 
reality technology. The importance of immersion has been seen in today’s world real 
life with tremendous changes and support services known as remote work and studies. 


Engagement and the Internet of Things on Human Psychology. The internet of 
things offers a unique opportunity to help empower humans and improve societies’ 
engagement. Empowerment improves the human way of reasoning, responsibility, and 
interaction. Engagement creates a digitalized altitude of dominant coalition motivation 
and the ability to deploy the resources. The ability of humans to feel relevant plays an 
important role in their way of reasoning and manners of doing things. In the healthcare 
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sector, human psychology is very important in determining health issues. A positive 
mindset helps the health practitioner to facilitate treatment which can be achieved by 
deploying virtual technology. 


Interaction and the Internet of Things on Human Psychology. The internet of Things 
helps create more significant learning spaces for human interaction in virtual systems. 
The ability to allow humans from a free physical background to run some modernized 
experience exercises is very important to human life and extremely shapes human psy- 
chology. Virtual objects are very important in advancing interaction for internet-based 
systems. 


Imagination and the Internet of Things on Human Psychology. The internet of 
things provides means for systems to surpass human imagination with good and positive 
systems. The context of the internet of things is becoming the number one system 
that is reinforcing organizations to rethink their value creation and methodology. With 
the emergence of the Internet of Things, it is easier and cheaper to make information 
available about virtual physical objects as this information can be automatically created, 
and distributed by simple stream of imagination. 


6 Conclusion 


The problem of content extraction is very essential in managing the amount of content, 
but other important issues associated with this are the viewpoint of the creator, quality of 
content, usability of content, and multiple views without changing the original content. 
This has not been the case in the past which is why this study applied deep learning 
to natural language processing using parts of speech to evaluate the changing nature of 
humans and the content of their information. Based on the behavior-oriented drive and 
influential function of the internet of things on human psychology “internet of thoughts” 
on content extraction, results show a score grade “Very Good” as per classification 
for both classes A and B but Class B is better than Class A with behavior score of 
4.58 and 4.09 respectively. The study concluded that the internet of things based on 
virtual reality technology influences human psychology “internet of thoughts (loThs)” 
on content extraction using natural language processing and deep learning models. This 
will help E-health services be able to accurately determine patient’s situations remotely 
and prescribe treatment. Digital objects like pointers, editor’s boards and working space 
should be part of Internet of Things with high visibility to health improve cognitive 
importance and promote healthcare and well-being of uses. 
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Abstract. This study aimed to develop and implement smart care spaces (SCS) to 
improve the quality of life and convenience for individuals with significant severe 
disabilities. By analyzing Korean and international case studies, the study inte- 
grated technology, universal design principles, and stakeholder feedback, along 
with care robots for daily living assistance. Various areas within the SCS address 
the unique needs of people with severe disabilities. The first SCS was constructed 
in the Nuri Hall of the National Rehabilitation Center in Korea as a pilot project. 
Future plans include continuous improvement and utilization of the space based 
on user experiences, feedback, and results of usability evaluation to reduce the 
caregiving burden and promote inclusive environments. 


Keywords: people with significant severe disabilities - care robots - service 
model - smart care space 


1 Introduction 


The rapid aging of the Korean population has led to an increasing demand for care ser- 
vices, while the younger generation’s population structure is decreasing at an unprece- 
dented rate. As the number of older individuals with disabilities continues to increase, 
there is an urgent need to address the care burden and determine effective solutions. 
This is particularly important since many severely disabled older adults and those with 
reduced mobility spend most of their time at home with caregivers, greatly impacting 
their quality of life. 

Previous studies investigating the living environments of people with disabilities 
[1-3] revealed that their living conditions play a significant role in determining their 
quality of life. In response, the Korean local governments have been implementing 
housing improvement projects for the disabled, such as installing residential convenience 
facilities for low-income registered persons with disabilities [4]. However, the creation 
of barrier-free spaces that allow for active and personalized care, accommodating both 
people with disabilities and their caregivers, is urgently needed. 
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As part of the 4th industrial revolution [5], the development of care robots [1, 6- 
8] has the potential to assist caregivers and reduce their care burden. However, the 
environmental factors need to be considered for the successful integration and utilization 
of care robots. Moreover, as Korea is projected to become a super-aged society by 
2026, the number of households requiring elderly care is expected to increase, further 
increasing the demand for home care solutions. 

To explore how care robots and services can be effectively implemented in vari- 
ous settings, we conducted a comprehensive analysis of the literature, field cases, and 
stakeholder interviews through a care robot translational research and service model 
development project (2019-2022). Based on our findings, we constructed a smart care 
space (SCS) on the second floor of Nuri Hall in the National Rehabilitation Center, 
Korea. This paper aims to discuss the design, implementation, and implications of this 
innovative SCS, which integrates technology and universal design for people with signif- 
icant severe disabilities, to address the care challenges in the care of the aging population 
with and into disabilities. 


2 Materials and Methods 


From January to December 2020, in order to develop the appropriate SCS for severely 
disabled older adults, results of the literature review of similar cases in Japan and Korea 
were analyzed, and various cases were collected by visiting the target sites that had a 
barrier-free design at home and abroad. A total of 11 institutions were visited to collect 
case studies. Next, 7 domestic cases such as that in Shinnae Medical Safety House and 
3 foreign cases such as that in Denmark were visited and benchmarked based on the 
collected data. Then the results were used to determine the SCS design. 

In this study, the British Design Council in 2005 adopted the double diamond process, 
which involves problem solving carried out through a four-step process (discovery, defi- 
nition, development, delivery), as a service design developed and analyzed the effective 
user experiences. 


e Discover: An in-depth interview was conducted to discover the opportunities provided 
by SCS and to analyze the users’ needs. On-site interviews were conducted in 3 pairs 
(45-year-old quadriplegic, 40-year-old spinal cord injury, and 45-year-old spinal cord 
injury, and 61-year-old, 60-year-old, and 58-year-old female caregivers, respectively). 
The needs for improving the user experience (UX) were determined. 

e Define: Through the creation of a persona that includes specific needs and pain points, 
the need for a point of contact between specific scenario-based service provision and 
individual goals was identified. 

e Develop: Based on the ideas derived from the goals of the customer journey map and 
interviews with experts, the technical feasibility was reviewed, and the composition 
of the service environment for this study was established. 

e Deliver: The service process reflecting the user’s life cycle was created as a situational 
scenario based on the research goals and user experience. 


In order to establish the SCS for the severely disabled, the areas that need improve- 
ments were determined through stakeholder meetings with the severely disabled and 
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caregivers. Interviews were conducted with experts, persons with severe disabilities, 
and caregivers. 


3 Results 


3.1 Survey of Current Status and Stakeholder Interviews for Smart Care Space 
Design 


Between 2019 and 2020, the appropriated SCS model was designed by determining the 
current situation and gathering expert opinions, related to the residential environments 
of domestic wheelchair users. We collected case studies carried out in various facilities 
based on a comprehensive literature review. 

Through stakeholder interviews surrounding Smart Care Spaces (SCS), the prospec- 
tive opportunities and users’ needs of care receivers and caregivers were analyzed. These 
needs are critical to be accommodated within their communal living space. Given that 
care receivers frequently spend the bulk of their day either in a supine position or require 
assistance, it’s imperative for them to be able to independently operate key devices 
or equipment. Simultaneously, caregivers showcased significant apprehension towards 
unfamiliar technology or devices, especially when utilized for the first time, underscoring 
the necessity for straightforward and intuitive user experiences. 

As such, the SCS should be envisioned as an interactive environment in which care 
receivers and caregivers cohabit and partake in daily life activities together. The focus 
should be on providing services that cater to individual needs and alleviate common 
inconveniences. 

Furthermore, it was found that the majority of Korean caregivers are older women 
with a generally low level of technological proficiency and a pronounced fear of 
new technologies. This demographic tends to react sensitively when interacting with 
technological devices, thereby necessitating an approach that is user-friendly and 
non-intimidating. 

In expert interviews, it was proposed that the incorporation of Internet of Things 
(IoT) technologies is crucial to enhancing user satisfaction and the overall quality of life 
in smart care spaces. These technologies could encompass voice recognition for simple 
on/off functionalities, sensor-driven alarm systems, behavior detection technology, and 
services that respond to user-intent. The main implication from this recommendation is 
the need for these technologies to be effortlessly usable and conveniently manageable 
by the end users. 

The different notable domestic cases that showcased our SCS designs were as fol- 
lows: Shinnae Medical Safety House, which features safety devices and space designs 
tailored to individuals with disabilities; Hanssem Smart Home Model House, which 
utilizes a Wi-Fi-based smart home system; Seongnam Senior-friendly Comprehensive 
Experience Center, Seongnam Senior Smart Home, and the Home Adaptation Train- 
ing Center of the National Transportation Rehabilitation Hospital, which emphasize 
the senior living spaces that incorporate advanced technology from the 4th industrial 
revolution era, with a focus on the Internet of Things (IoT) technology; and the Korea 
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Institute of Science and Technology (KIST), Gachon University Ambient Assisted Liv- 
ing (AAL), and Bundang Seoul National University Hospital, which offer separated 
spaces for experiments, exhibitions, and monitoring, as well as for usability evaluations. 

Additionally, we benchmarked several international cases that showcased innovative 
solutions not yet implemented in Korea. These included the installation of folding fur- 
niture attached to the wall and wall mounted-type lifts that provided assistance during 
spatial movement at Musholm (Denmark, a residential facility for people with muscular 
disorders); the application of universal design principles to enhance usability at the Life 
and Living Center (Helsinki, Finland); and SilviaBo (Sweden, a residential facility for - 
older adults with dementia), which had simplified spatial structures and improved indoor 
circulation for wheelchair users. 


3.2 Smart Care Space Design 


Main Entrance and Room Door. Wheelchair users find it easier to operate sliding 
doors rather than pull-style doors. Although the main entrance door features a set of 
double pull-style doors for ease of installation, all room doors have been designed with 
sliding door systems. When designing entrances and corridors, the required effective 
width and front effective distance should be considered. For bedroom access, the effective 
passage width, activity space, bed structure, and flooring should be taken into account. 
Installing semi-automatic or automatic sliding doors at the entrance is recommended, 
along with securing activity space in front of and behind the entrance to avoid collisions 
with hallway pedestrians. To accommodate wheelchair users, a toilet door should have 
an effective width of at least 0.8 m, a step of 2 cm or less, and a sliding door rather 
than a hinged one, as this provides more activity space in front of the door and added 
convenience. 


Living Room and Hallway. Architectural considerations should be made to ensure a 
clear and obstruction-free pathway for wheelchair users. Since it can be challenging for 
wheelchair users to move in a straight line when their left and right manipulation power 
is not equal, a wider walking width is necessary. Power wheelchairs require a width of 
800 to 900 mm, and the rotation widths should be 2,000 mm for power wheelchairs and 
1,800 mm for manual wheelchairs. By securing an aisle width of more than 1,200 mm for 
emergency beds, placing safety handles in corridors, and installing unit households with 
emergency alarm lights at entrance tops, emergency contact devices inside households, 
and motion detectors, safety can be ensured. 


Restroom. To prevent head injuries, a sufficient distance should be maintained between 
the toilet and the wall, and wheelchair users should have access to either side of the toilet 
depending on their dominant hand. The effective toilet size should measure at least 1.4 m 
in width and 1.8 m in depth, with a 0.75 m or wider activity space secured on one side 
of the toilet for side wheelchair access. The toilet seat height should be between 0.4 m 
and 0.45 m from the floor, and an adjustable wall-mounted toilet should be installed 
to accommodate wheelchair users, older adults, and children. A user-friendly flushing 
mechanism, such as a light-sensing push-button or lever type, should be installed. 
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Others. The height of wall hangings, electrical outlets, switches, storage shelves, and 
hangers should be between 0.8 m and 1.2 m from the floor; if a bed is not used, the instal- 
lation location of electrical outlets and switches should be within 0.35 m. The doorbells 
for hearing-impaired disabled individuals must be installed in rooms, bedrooms, and 
bathrooms. The floor surface of all spaces should be level, be finished with non-slip 
material even when wet, and have an effective floor area of more than 1.4 m in width 
and 1.8 m in depth. 


3.3 Design and Construction of SCS 


Design of SCS. Based on the literature review and current status survey, an initial 
SCS design draft was developed and refined using the input from architectural experts 
(Fig. 1A). Design drawings were then used to gather and apply the following field case 
characteristics (Fig. 1B): 1) incorporating double doors (final width of 1,600 mm) of 
at least 1,100 mm at the main entrance for easy movement of emergency beds and 
wheelchairs; 2) designing steps within 20 mm at the entrance to facilitate wheelchair 
movement; 3) removing steps at the bathroom entrance and installing a water drainage 
trench to prevent water from flowing back into the main room; 4) using wheeled fur- 
niture or beds for easy mobility; 5) maximizing space utilization with wall-mounted, 
foldable furniture; 6) positioning ceiling-mounted lifts, height-adjustable washbasins, 
and foldable shower beds for improved bathroom usability; 7) creating a multipurpose 
room near the entrance for lift storage; 8) arranging a straight-line wall-mounted lift 
for convenient bathroom access by connecting the toilet to the main room; 9) designing 
a barrier-free entrance and toilet considering the turning radius of electric wheelchairs 
(more than 1,500 mm); and 10) installing sliding bathroom doors instead of hinged doors 
for optimal space usage. 


Fig. 1. Smart care space construction design (draft). (A) Space size and design draft, (B) Smart 
care space final design draft 


Establishment of Smart Care Space (Nuri Hall, National Rehabilitation 
Center). The National Rehabilitation Center’s Smart Care Space was established as 
part of the Ministry of Health and Welfare’s care robot translational research and service 
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model development project, which started in 2019. The space, built for demonstrations 
like usability evaluations of care robots or care devices, is the first in Korea to address 
the needs of both severely disabled individuals and their caregivers. The initial SCS 
was constructed in the National Rehabilitation Center’s Nuri Hall (December 2019) and 
opened in 2020 (Fig. 2). It integrates care robots and various sensor-based IoT facilities 
and devices into six spaces to create a convenient and safe care environment that accom- 
modates various care scenarios and physical conditions, emphasizing IoT technology 
and universal design features. 


The SCS was designed to harmonize with the needs of severely disabled individuals 
and their caregivers, thus enabling care robots to assist with daily tasks seamlessly. It 
also serves as a demonstration space for caregiver burden analysis and care robot usabil- 
ity evaluations. The service environment for severely disabled individuals includes a 
Google speaker and application, functioning as the central controller in the bedroom, 
which connects to other devices in the house and provides personalized services. Voice 
commands such as “Come out” and “T11 go out” can sequentially control various devices, 
while natural language voice commands enable users to manage air-conditioning, light- 
ing, and weather information on-demand. The UI and UX were designed to facilitate 
device control. 


Bath room 


Entrance Living room Kitchen Dining room Bed room 


Fig. 2. SCS built in the National Rehabilitation Center, Korea (Photos of 6 spaces of SCS). SCS: 
smart care space. 


4 Discussion 


The SCS offers a revolutionary approach to create tailored, efficient, and accessible 
care environments for people with significant severe disabilities. First, it integrates 
cutting-edge technology, universal design principles, and valuable stakeholder feedback 
to address the unique needs of and challenges faced by both disabled individuals and 
their caregivers. The incorporation of the barrier-free concept in the SCS, enables the 
efficient use of care robots, thus enhancing the effectiveness of this innovative space. 
Second, as a model for future development, the SCS aims to improve the quality of 
life for those with disabilities and reduce caregiver burden through the incorporation of 
care robots and other assistive devices. By continuously evolving based on user experi- 
ences and feedback, the SCS can potential transform the care landscape, fostering more 
inclusive and supportive environments for disabled individuals and their caregivers. 
The Nuri Hall of National Rehabilitation Center’s SCS allows the utilization of care 
robots that provide technical assistance in caregiving and enhancing care quality, while 
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also serves as a demonstration space for care robot prototypes. Considering the various 
stakeholders’ input, the SCS integrated technologies related to daily living convenience, 
such as barrier-free design, smart controllable IoT devices, voice control applications, 
and wall-mounted transfer devices tailored for wheelchair users. Opened in 2020, the 
space caters to the needs of severely disabled individuals, primarily wheelchair users, by 
designing and organizing spaces based on daily life scenarios illustrated through journey 
maps. It also features pressure ulcer prevention and posture-changing beds, excretion 
care robots, and meal assistance devices. 

The SCS have been designed and built to enhance the provision of beneficial services 
in a care environment by integrating networked care robots that can interact with indi- 
viduals. In light of this, there is a pressing need to explore methodologies for the joint 
expansion and provision of care robots and care environments, particularly for those 
most vulnerable, including people with severe disabilities, older adults with notably 
limited mobility, and caregivers in Korea. 


5 Conclusions 


The SCS’s most distinguishing feature is its capacity to serve as a genuine residential 
environment, unlike many living labs or experience centers in Korea. The active par- 
ticipation and choice of users in the design process play crucial roles in the discovery 
and innovation of new ideas [9, 10]. In the future, patients at the National Rehabilita- 
tion Hospital, caregivers, and their families can directly use the space, enabling them 
to implement similar scenarios and features within their communities. This study does 
have some limitations. First, the SCS was designed in alignment with Korea’s barrier- 
free housing blueprint, with specific reference to the living spaces of severely disabled 
individuals. Moving forward, there is a clear need for research that gathers data from the 
actual residences of a diverse range of users. Second, it is essential to assess the usability 
of the SCS across users with various disabilities to ensure inclusivity and accessibility 
in its truest sense. The SCS model will provide knowledge and guidance while con- 
tinuously improving the space through the user’s feedback on areas for enhancement 
after actual residency or usability evaluations, ultimately demonstrating a reduction in 
caregiver burden. 
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Abstract. In our research, we established a medical annotation model in the form 
of an ontology in an effort to ensure data interchange amongst medical annotation 
systems. We employ the “patient partner” approach to involve the patient in the 
medical annotative activity. In fact, the patient will be able to register, annotate, and 
comprehend the comments made in his medical file utilizing this new paradigm. 


Keyword: Annotation - Medical record - Interoperability - Patient - Ontology 


1 Introduction 


Over the last two decades, health information systems have experienced a significant 
technological revolution, which has affected the quality of services provided by these 
systems, their functionalities, the extent of their applications, and the sorts of users that 
use them [1]. 

As aresult of this transformation, the Electronic Medical Record (EMR) is one of the 
components of medical information systems that are being transformed. It is true that with 
the use of ICT (information and communication technologies) in the medical area, the 
medical record, which was previously on a physical medium, has now been converted to 
a digital version. The latter is defined as an electronic statement that contains a summary 
of treatments, diagnoses, and patient follow-ups, as well as any written communications 
between healthcare specialists [2]. 

Currently, scientific research is seeing a slew of initiatives to include the patient in the 
decision-making process along his treatment course. This is beneficial to the healthcare 
worker, the patient, and the whole healthcare system because it encourages collaboration, 
cooperation, and knowledge exchange, and it contributes to an improvement in the safety 
and quality of treatment. Patient participation in the healthcare system is focused on a 
variety of levels, including acquiring medical knowledge, educating, establishing the 
treatment protocol, providing direct care, organizing care, and so on [3]. 
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In this study we will investigate the applicability of the patient partner approach 
to the area of computerized medical record annotation. Indeed, traditionally held and 
examined solely by a medical team in a single health establishment, local electronic 
medical files are now shared and consulted by new interveners, the hosts, and the patients. 
An important part of the host’s job is to create an application that includes both patient 
care and data preservation. Electronic medical records cannot be created or accessed 
without a patient’s agreement or authorization. Furthermore, he is entitled to see his 
medical records as well as papers pertaining to the diagnosis and treatments that are 
relevant to him. In this sense, the patient should also have the right to comprehend the 
annotations that have been made to his medical record. 

A medical annotation is, in reality, a comment that is written in the patient’s medical 
file for a particular purpose. In this case, it is done by the healthcare practitioner and is 
meant to be read by another healthcare professional or by the patient themselves. As a 
result, we can say that medical annotation is a fundamental part of the healthcare system. 
She is the one who encourages open dialogue and information exchange among the many 
parties involved. However, a review of the literature on numerous annotation systems 
revealed that all of these techniques remain basic and are not widely used in the health 
system [4]. Patients are unable to access the content of the annotation systems examined 
in the literature, they are unable to communicate their ideas in a meaningful manner, 
and as a consequence, the annotation is unable to promote collaboration among various 
stakeholders. It will cease to work properly, resulting in a breakdown of communication 
between healthcare practitioners and patients. 

Each annotation system investigated in the literature has its own set of standards and 
conventions. The inability to integrate diverse programs prevents data interchange across 
applications. Due to trade blockage, a healthcare professional cannot communicate or 
receive data from peers or transfer information across systems. This restricts the move- 
ment of both patients and healthcare providers. Also, the patient cannot send personal 
data to his doctor or a health practitioner, resulting in data loss. Indeed, this highlights 
the need to address the challenge of interoperability among different annotation systems. 
The objective of this work is to develop an ontology that ensures the semantic layer of 
interoperability. 

So, what follows is an attempt to answer a fundamental question: 

What generalized model of medical annotation should we use to promote health 
professionals and patients to exchange annotations in electronic medical records? 

The second section of this paper provides a comprehensive understanding of the 
different types of medical records and highlights the crucial importance of facilitating 
their sharing. 

The third section delves deep into an extensive exploration of the diverse definitions 
of annotation found in the literature. This thorough analysis captures a wide range of 
perspectives offered by researchers in the field. 

The fourth section introduces a meticulously developed ontology specifically 
designed for layered annotations within the medical record. This ontology serves 
as a robust framework that ensures semantic interoperability across various medical 
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annotation systems. By standardizing the annotation process, this ontology signifi- 
cantly enhances data exchange and promotes seamless collaboration among healthcare 
professionals. 

The fifth section is a conclusion that presents a comprehensive summary of our 
findings while also providing valuable insights for future research in the domain of 
medical record annotation. 


2 Computerization of Medical Record 


e Medical record sorts 

A patient’s medical record is defined as a statement that contains a summary 
of treatments, diagnoses, and patient follow-ups, as well as any written interactions 
between health care providers. 

Examining various forms of medical data gives critical information for medical 
annotation modeling. It makes it easier to interpret medical data, identify specific 
annotation requirements, build suitable annotation schemas, and validate annotations. 

In this part, we will look at the various forms of medical records and highlight 
their key advantages. 

e The EMR Electronic Medical Record and the CCR Computerized Clinical 
Record: 

The EMR and CCR are electronic equivalents of the traditional medical record, 
which is recorded on paper with a pencil. They are memories that allow the acquisition 
of personal data from patients [Article 1 of the Canadian Uniform Act on Electronic 
Commerce]. These files are deemed complete since they contain all of the essential 
information and specialized words relevant to a given speciality and also create the 
proper instruments for trade practice. 

e The EMR Electronic Medical Record and the EHR Electronic Health Record: 

The EMR is distinguished from the EHR by the Canadian Medical Protective 
Association (CMPA)!. The EHR delivers complete information from many providers 
across numerous provinces and regions. Nevertheless, this sort of record provides less 
detail than the EMR (it does not contain all the information presented in the EMR). 
The concept of electronic record sharing arises from the capacity of a health care 
practitioner to adhere to data generated and kept by someone else. As a result, he 
may study them and use them within the context of his specialty’s practices. He can 
also contribute information to it. We can look at Quebec, where the QHR (Quebec 
Health Record) was implemented, and France, where the SMR (Shared Medical 
Record) was constructed. The medical records implemented in France and Quebec 
are concrete examples that illustrate initiatives for sharing medical records, aimed at 
improving care coordination and patient management. 


l Les dossiers de santé électroniques: perspectives de la responsabilité médicale. 
Aout 2008. p. 5. http://www.cmpa-acpm.ca/cempapd04/docs/submissions-papers/pdf/com- 
electronic-health-records-f.pdf. ousur www.cmpa-acpm.ca. http://www.cdpdj.qc.ca/publicati 
ons/memoire-pl-59-renseignements-sante.pdf. 

2 ASIP Santé, Rapport d’ activités 2010. PP 26 et 28. http://esante. gouv.fr/sites/default/files/ASIP- 
RA2010.pdf. 
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e The local EMR and the shared EMR: 


Local EMRs are used and preserved only by its authors and are entirely their 
responsibility. Yet, more healthcare personnel have access to shared EMRs, which 
explains the rise in the number of stakeholders and managers. 


The various kinds of medical records are contrasted in the table below (Table 1). 


Table 1. Kinds of medical records 


EMR CCR QHR SMR 
Owner Medical clinic | Health institution | Service points Beneficiaries of health 
participating in insurance 
the DSQ project 
Content Information Information Results of Results of biology and 
recorded by the | recorded by the | examinations or | radiology reports, 
clinic’s clinicians of this | laboratory general medicine, 
clinicians at the | establishment at | analyses, results | treatments/prevention, 
time of the the time of the of imaging exams | certificates and patient 
consultation or | consultation or | and medication area 
care episode the episode of 
care 
individuals | Clinicians and | Clinicians and Authorized The patient chooses 
withaccess | anyone anyone professionals, which health 
rights authorized by authorized by the | Each authorized | establishmentor which 
the patient patient professional does | doctor will be able to 
not necessarily have access to his file 
have access to all | to consult or supply it 
parts of the file 


e Shared medical record actors 


Local electronic medical records, which were previously held and accessed solely 
by a team at a single health facility, are now shared and consulted by new stakeholders, 
the hosts and the patients. 

Host: The Host’s functionality entails creating an application that combines 
processing and preservation of patient data*-*. 

Patient: the establishment, accommodation, and access to the SMR or QHR 
cannot take place without the patient’s authority and agreement. The latter has the 
right to see his file, as well as the records pertaining to his diagnoses and treatments, 
and even to obtain information pertaining to his file. 

Healthcare professionals involved: The public health code mandates that med- 
ical practitioners get the patient’s permission before consulting their shared EMR. 
Within the confines of the professional practice of their trade, they are accountable for 


3 Article 19, de la loi sur les services de santé et les services sociaux. 
4 Article L 1111-21 du code de la santé publique. 
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any confidential information pertaining to the patient. So, they must keep all sensitive 
information hidden. 

e Contributions of medical record sharing 

The digital support relating to the medical file is an effective tool which contributes 
to facilitate the research of the information concerning the patients, to help to process 
the data and to ensure the storage. 

The table opposite (Table 2) shows the contribution of sharing medical record on 
an individual and collective level [5-7]. 


Table 2. Benefits of medical record sharing 


Individual benefits 


The patient’s right to view his medical file: 
the patient’s education and culture have a 
major impact on his consent and fear about 
his medical care 

Decision support and medical mistake 
prevention: Decision support systems are 
intended to give clinicians convenient 
access to test-based medicine and guideline 
treatments (automated vaccination 
reminders, for example) 


Automation of tedious tasks 


Collective benefits 


e Ease of information retrieval: for example 
the indexing of images by their content and 
the judicious processing of distributed 
heterogeneous information 

e Data and information processing: the 
medical record includes capabilities for 
graphically summarizing the patient’s state 
and illustrating the patient’s data in the form 
of complex signs 

Storage: Paper documents can only be kept for 

50 years, thus it’s critical to develop the 


software required for medical data storage 

e Decreased access, routing, and data sharing 
times: When several people have access, 
sharing and exchanging information is made 

simpler 


Security and traceability needs: 

the law of traceability requirements for access 
to information, data security and protection of 
individual freedom 


In this sense, we can state that the share of patient file ensures the efficient operation 
of the care protocol, helps to raise the standard of medical care, ensures data security, 
and helps stakeholders coordinate while enabling data access. Also, by allowing the 
patient access to his medical record, it is feasible to give him detailed explanations of 
his condition and course of treatment. As a result, both the professional and the patient 
have a vested stake in the outcome. 


3 Annotation of Medical Record 


Having a precise understanding of what constitutes an annotation is crucial for accu- 
rately describing an annotation ontology. In fact, the definition of annotation has sparked 
diverse perspectives among researchers regarding this concept. These varied viewpoints 
have played a significant role in enhancing our comprehension of annotation and have laid 
the groundwork for constructing a pertinent ontology that encompasses the concepts and 
relationships associated with annotation. As a result, the annotation ontology embodies a 


274 Z. Mannai et al. 


comprehensive understanding of the annotation concept by taking into account multiple 
researchers’ perspectives. 


e Annotation is also referred to as “the trace of an action” by researchers. Annotations 
show that the person who reads a text is engaged intellectually and makes an effort 
to annotate it [8, 9]. 

e An annotation may also be thought of as “a visual shape” In fact, an annotation must 
be visible in order to be deciphered [10]. 

e Annotation is a “visual shape with a purpose”. It is critical that the annotation 
serves a function. Nobody annotates for no reason. There is always a motivation for 
the annotator to annotate.[11] 

e Annotation is defined by some scholars as “a trail of activity in a visual form with 
a purpose” Therefore it is a record of the annotator’s mental representation on a 
document for a specific purpose [12]. 

e The annotation is “a visual shape connected to a page” according to [13]. According 
to this scholar, an annotation must be tied to a document at all costs since if the paper 
is lost, so is the annotation. 

e In the discipline of computer science, we must employ an anchor to describe the 
placement of the annotation in the document since “the annotation is a visual form 
and an anchor”. The annotation is defined in this same field as “visual form attached 
by an anchor with a right of access” Only those who are permitted may annotate a 
document in this situation [14]. 

e An annotation, according to some academics, is “a trail of action in a unified visual 
form coupled with an anchor” The mental image that the annotator has created 
about the target is the annotation, and we use the anchor to tie the annotation to the 
document [15]. 


4 Medical Annotation Model 


Our objective is to propose a formal model of annotation to aid in the resolution of issues 
that arise while researching annotation systems [16—18]. 

In 2001, the W3C released the initial draft of the Annotea standard, which aimed 
to define an annotation model based on the RDF representation. Over time, this model 
underwent significant enhancements by the dedicated research team, eventually leading 
to its official adoption as a standard by the W3C in 2014. Despite the evident success of 
RDF technology in the semantic web domain, upon which Annotea is built, the adoption 
of Annotea for annotating systems development has been limited among researchers. To 
address this issue, researchers [19] developed an ontology that drew inspiration from the 
Annotea model. We utilized this ontology as a fundamental basis and further refined it 
to tailor it specifically to the medical field. 

Our ontology development followed the established methodology employed at Stan- 
ford University, which encompasses seven fundamental steps: Domain definition of the 
ontology, Reuse of existing ontologies from the literature (if available), Definition of the 
set of important terms, Definition of classes and their hierarchy, Definition of properties 
of the classes, Definition of attribute facets, and Creation of instances. By following 
this methodology, we ensured a systematic and comprehensive approach to ontology 
development for our research [19]. 
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To effectively illustrate the foundational concepts of our ontology, we can take the 
example of a radiologist who wants to annotate the medical field of his patient. The 
radiologist, as an annotator belonging to the medical team, has the patient’s consent to 
access their medical record and make annotations. On a specific date, in a particular 
healthcare facility, using their PC, the radiologist wishes to write an annotation on a 
specific paragraph of the patient’s electronic medical record. The radiologist uses their 
PC and connects to the healthcare facility’s medical information system, which contains 
the patient’s electronic medical record. 

While navigating through the record, the radiologist locates the paragraph on which 
they want to write an annotation. To attach the annotation to the target paragraph, the 
radiologist uses an anchor, which can be a feature provided by the medical information 
system or a dedicated annotation tool. They select the relevant paragraph and access the 
section reserved for scientific annotations within the document. In this section for anno- 
tations, the radiologist writes their annotations to capture the important points. They may 
provide additional information, observations, interpretations, preliminary diagnoses, or 
other relevant information related to the content of the paragraph. The annotations are 
saved in the patient’s electronic medical record, typically in a structured format and 
associated with the corresponding anchor, thereby linking the annotation to the specific 
paragraph. The objective of this annotation is to retain the important points and pro- 
vide additional information for future reference. This can help other healthcare profes- 
sionals understand the radiologist’s reasoning, clinical decisions, and evaluations when 
reviewing the patient’s medical record. 

The proposed ontology includes the following concepts: 


e Physical attribute 

Anchor: An annotation’s anchor is a way for an annotator to define where the 
annotation should be placed within the document. 

Content: it is a trace of the mental representation that the annotator has developed 
concerning the object of interest. 

Target: the target serves as the basis for the annotation. The healthcare profes- 
sional has the possibility of placing his annotation either in the section reserved for 
scientific annotations, or in the section reserved for adopted annotations, as the case 
may be. The patient can keep his notes in a part that has been reserved for him. 

The following figure highlights the subclasses of the physical attributes class 

e Circumstantial attribute 

Circumstantial attribute encompasses all aspects of contact with the environment 

(date, place, tool, validity). 
e Partnership Attribute 

These characteristics arise when many types of stakeholders collaborate together 
on a project. 

Intervenor: a person who has the authorization of the patient to consult and 
annotate his electronic medical record is called an intervenor. 

Medical team: in this case, the speaker is a member of the medical profession. 

Non-medical team: the intervener, in this case, is not a health professional. 

Patient: the patient is a practitioner who has the power to take notes in his medical 
file. 
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Patient: The patient is entitled to view and make notes in his medical record. 

Host: it is the role of the host to create an application that includes both the 
processing and archiving of patient data. 

Patient’s family: Electronic medical records (EMRs) can be viewed and 
annotated by a patient’s family members with the patient’s permission. 

Level of analysis: The term “level of analysis” refers to the process of analyzing 
the extensive data available on the patient. 

Partnership level: the different stakeholders must collaborate in order to reach 
a consensus on a decision. For this reason, they are required to go through the four 
degrees of partnership. 

Transmission level: this level signifies the beginning of the discourse phase 
between the different parties involved in the process. 

Planning level: at the planning stage, all the different stakeholders work together 
to plan each step of the care pathway that the patient will go through. 

Action level: it is about going to the act of care and implementing all the activities 
previously prepared. 

Informative unit: an informative unit is a collection of information on an illness 
or medical event that pertains to a patient. 

Quantifiable piece of information: for example, the number of hours slept or 
meals consumed during the course of a day. 

Qualitative piece of information: patients’ emotional levels or their general 
health information are examples of qualitative data. 

Personalization parameters: they refer to the characteristics that distinguish 
one patient from another in terms of reading style and annotation generation process. 
Personalization parameters, as stated in earlier sections, are as follows: (learning 
level, media preference, language preference, specific requirements, beliefs, social 
norms, psychological data). 

Access authorization: the authority required for a user to have access to protected 
data or resources in a medical record concerning a certain patient. 

e Semantic Attribute 

Semantic attributes are the characteristics which make it possible to adapt the 
annotation to the function to which it applies. They help make sense of annotations 
by providing context. 

Perlocutionary attribute: an annotation is produced for a specific purpose by 
the person who makes it. 

Production objective: the objectives of the annotator with reference to the 
creation of his note. 

Reading objective: these are the annotator’s expectations for reading his note. 

Communication object: the communication object identifies the subject of the 
annotation, ie the subject of reading the annotation. 

Reuse context: the reuse circumstance is a representation of how the annotation 
was intended to be reused. 


The (Fig. 1) illustrates the proposed ontology’s class hierarchy. 
The class hierarchy view displays asserted and inferred class hierarchies. By default, 
the stated class hierarchy is shown. The asserted class hierarchy view is one of the 
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Fig. 1. Class hierarchy 


primary navigational features in PROTEGE. It is displayed as a tree, with the nodes of 
the tree standing for the classes. Sub/super class relationship in the class hierarchy is 
represented by a child-parent relationship in the tree. 

In our ontology, we have the class “Medical annotation” is the parent class of the 
classes “Physical attribute”, “Semantic attribute”, “Circumstantial attribute” and 
“Partnership attribute” which are sister classes that appear under their parent class 
“Medical annotation “. 

“Anchor”, “Target” and “Content” are sister classes derived from the “Physical 
attribute” class. “Anchor” is the parent class of the “Scientific annotation”, 
“Adaptive annotation” and “Patient annotation” classes. “Communication object”, 
“Perlcutionary attribute” and “Reuse context” are equivalent classes. They are the 
daughter classes of the “Semantic attributes” class. 

The “Partnership attribute” class is parent class of the classes: “Partnership level’, 
“Access permission” and “Stakeholder”. “Stakeholder” derives into two classes which 
are: “Medical team” and “Non Medical Team”. “Host”, “Patient’s family”, “Patient” 
are the daughter classes of the “Non medical” class. “Transmission level’, “Analysis 
level’, “Planing level” and “Action level” are the subclasses of the “Partnership level” 
class. 

(Figure 2) showcases the object properties within the proposed ontology, providing 
a visual representation of the relationships established between entities. 
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Fig. 3. Data properties 
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(Figure 3) provides a visual representation of the data properties integrated into the 
proposed ontology, illustrating the specific attributes and characteristics associated with 
the entities. 

(Figure 4) presents an example axiom for the class ‘MedicalAnnotation. This axiom 
specifies that instances of the class “MedicalAnnotation” must have at least one “Partner- 
shipAttribute’”, at least one “SemanticAttribute’’, at least one “CircumstantialAttribute”’, 
and at least one “PhysicalAttribute”. The quantifier (some) indicates that there exists at 
least one instance that satisfies the restriction. 


@ HasAsCircumstantialAttribute some Circumstantial_attribute 
@ HasAsPartnershipAttribute some Partnership_attribute 

@ HasAsPhysicalAttribute some Physical_attribute 

& HasAs SemanticAttribute some Semantic_attribute 


Fig. 4. Example of axiom 


5 Conclusion 


In conclusion, the development of an ontology for annotation in the medical domain 
serves the purpose of creating a standardized and structured representation of medical 
annotations. By capturing the knowledge and semantics associated with these annota- 
tions, the ontology enables easier utilization, sharing, and interoperability across diverse 
applications and systems. The proposed ontology undergoes discussions and valida- 
tion by domain experts, and it is essential to emphasize that the validation of ontology 
performance in medical applications is an ongoing process. User feedback and regular 
evaluations play a crucial role in identifying areas that require improvements and allow- 
ing for adjustments to be made to the ontology accordingly. As part of future work, we 
plan to implement an annotation system based on this ontology, further enhancing the 
efficiency and effectiveness of medical annotation processes. 
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Abstract. Measuring the range of motion (ROM) is one of the important tasks in 
medical or healthcare sectors. However, person-to-person measurement is time- 
consuming and requires resources. In this paper, we propose an approach to esti- 
mate ROM using machine learning algorithm equipped with computer vision tech- 
nology based on data-driven experiments. We describe the setup to gather experi- 
mental dataset to learn the angle of human joints in 2D space. From the extensive 
experiments and multi-linear regression learning approach, our method can reduce 
estimation error by 11.1% in average. We draw conclusions that machine learning- 
based, data-driven approaches could predict ROM better than using vision camera 
only. 


Keywords: Range of Motion - Machine Learning - Multi-camera - Multi-linear 
Regression - Rehabilitation 


1 Introduction 


The measurement of the range of motion (ROM) is critical in health care as well as 
clinic treatment professionals. Also, the importance of ROM is increased in the field 
of rehabilitation exercises (RE) from surgery, disability, accidents and so on. The mea- 
surement devices of ROM (MDROM) can be applied to yield ROM information only if 
ROM values are measured by well trained and experienced persons with relatively high 
precision. However, P2P measurement with MDROM has several drawbacks. First, P2P 
with MDROM needs resources such as education, training for professional knowledge 
which follows cost and certain period of time. Second, it has constraints since at least 
two persons must be met in same time and physical space. Third, even though a person 
who measures is well trained, the results are not always be consistent since it depends 
on the persons. 
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With the rapid progress of artificial intelligence (AI) in image processing recently, 
there is a possibility that these shortcomings can be compensated. AI bears various areas. 
One of the Al-based image processing areas is the detect pose landmarks in an image or 
video streaming. MediaPipe provides open application programming interfaces (API) 
[3], called ‘ML kit’ to software developers for the purpose of ease and fast building 
applications. Right figure of Fig. 1 is an example figure provided by ‘ML kit pose detec- 
tion API’ (we will use ‘ML-kit’ for brevity hereinafter). ML-kit provides 33 coordinates 
of body landmarks including entire body (face, arms, legs, etc.) For each coordinate is 
consisted of three values representing x-, y-, and z-axis of a landmark. 

Image processing powered by machine learning algorithms show the powerful abil- 
ity to replace P2P MDROM. Also, researchers show practical implementations in var- 
ious areas such as underwater running [4], marker less system [5], pose matching [6], 
measuring single-leg squat kinematics [7] and so on. 


À ` 


Fig. 1. Two examples for measuring human body. P2P based ROM measurement (left) [1] and 
Machine learning based pose (landmarks) detection [2]. 


Besides medical and rehabilitation, RE machines are already on commercial mar- 
ket. For examples, multi-purpose device including rehabilitation, therapy and fitness is 
introduced in [8], mobility-abled rehabilitation exercise equipment [9], rehabilitation 
machine for people with disabilities [10], home healthcare device with IoT [11], and so 
on. However, almost RE doesn’t have functionalities for ROM using computer vision 
systems. We argue the main reason is that output data from computer vision is not reliable 
and hard to implement onto RE in terms of accuracy compared to P2P approaches. 

We focus on the machine to be used in RE using machine learning to facilitate 
ROM with vision systems in this paper. We developed new RE machines during our 
research project mainly focused on upper body rehabilitation exercise/training machine. 
To exploit the computer vision systems, we installed camera devices and developed 
analysis software for various motions in rehabilitation exercise. One of the challenging 
problems is that since the images from rehabilitation training is 2D-coordinate, it is 
not trivial to estimate body angles in actual 3D-coordinates in real world. The easiest 
approach is to apply multiple cameras to measure ROM on RE. However, mounting 
several cameras is disadvantageous in terms of RE operation, maintenance, data pro- 
cessing of computing devices inside RE, and movability to elsewhere. We addressed the 
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problem-solving approach utilizing machine learning to overcome insufficient 2D space 


information in 3D world. 


2 Problem Definition 


2.1 Problems of 2D Information in 3D World 


To give better experience and systemic rehabilitation exercise, we developed a RE, which 
can monitor, and analysis 13 types of motions based on computer vision system. We 
report the possible rehabilitation actions of RE we developed in Table 1. The RE is 
consisted of exercise devices, information screen, computing devices, and a camera. 


The detailed figure is shown in Fig. 2. 


Table 1. Supported rehabilitation motions in our RE. 


No Description No Description 

1 Left arm lateral up and down Head lateral up and down left 
2 Right arm lateral up and down 9 Head lateral up and down right 
3 Left arm anterior un and down 10 Lat pull down 

4 Right arm anterior un and down 11 Behind neck pull down 

5 Left elbow anterior up and down 12 Seated draw 

6 Right elbow anterior up and down 13 Chest press 

7 Head anterior up and down 


Camera 


Display 


Electronic & 
3 Computing 
Unit 


Fig. 2. The configuration of the RE we developed. A camera is installed on head area of RE 


We utilized ML-kit to detect landmarks of human body, which supports to extract 
3D coordinates of 33 landmark points in real time. For more detailed information, refer 
to [12]. Even though the ML-kit produce 3D-coordinates with the tuple values, (x, y, z). 


Enhancing the Measurement of the Range of Motion 285 


The x and y are the landmark coordinates normalized to [0, 1], and z is the depth at the 
midpoint of hips. 

Problems arises here is that if we consider only 2D image, the estimation of ROM 
has errors by nature. For example, if we measure angle of in shoulder-elbow-wrist of 
ROM in left elbow anterior up and down (Motion # 5 in Table 1) using only one camera 
as shown in Fig. 1, the angle is equal to 185°. However, if we measure the angle after 
moving the location of the camera to the left-side of the person the value of angle is 
equal to 118° which shows not acceptable errors. For the intuitive understanding, we 
portrayed the problematic situation in Fig. 3. 


Fig. 3. The landmarks from front camera (right) and those from the left side of person 


185 and 118° from the left- and right-hand figure of Fig. 3, respectively. Note that 
we applied well known mathematical equation to compute angle between three points 
in 2D space stated in Eq. (1). 


l radians 
n, — : 
angle = | 780 
—b -b 
radians = arcan( È >) -_ arcan( 2 >) (1) 
Cx — by ay — Dy 


where a, b, c are the 2D coordinates such that a = (ax, ay), 

b= (bx, by), c = (Cx, Cy) and |-| is absolute function, respectively. 

This example is just for one situation among 13 rehabilitation motions. To mitigate 
errors under information loss between the 2D and 3D space, we introduce machine 
learning approach in this paper. 


2.2 Problem Formulation 


We restrict the problem such that it is just for one situation among 13 rehabilitation 
motions. To mitigate Given a dataset D, we want to learn how to predict more precise 
coordinates information to estimate ROMs. As our notation, we define observed dataset 
D = {(9;, yD 1> Where o; is observed information and expressed by vector space, 
y; is label (true value), (-) is tuple containing a pair of data, N is the total number of 
data tuples, respectively. The maximum dimension of our data is R!*7*, consisting of 
24 = 8 points (landmarks) x 3(x, y, z). Also, we select the first step of this problem 
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applying simple machine algorithm, multiple linear regression (MLR), which is simple 
but fast learning algorithm. We will use root mean squared error (RMSE) to evaluate 
our approach, which can be stated as Eq. (2) 


1 x 
RMSE = —y/(yi — 5), (2) 


where ¥; is the predicted value by MLR. 


3 Experimental Setup 


To gather D, we conducted experiments for each rehabilitation motions as stated in 
Table 1. The experiments are conducted via 13 test subjects (refer Table 2). Each test 
subject performs 5 times for each rehabilitation motion. 


MM distance (cm) 
MEE the height of camera from ground 


110cm 


125cm 


44cm 
(test subject seated in wheelchair) 


Fig. 4. Camera positions in experiments, where the numbers on solid black lines are distances 
between a camera and test subject, the red numbers beside cameras are the heights of cameras 
from ground, the figures inside of yellow circles are the camera indexes. 


To gather various possible combinations of camera position, we installed four addi- 
tional cameras with different distances, heights, angle at which looking at the test subject. 
Finally, we have five camera configuration including rehabilitation measuring equipment 
(marked as ME in Fig. 4). We simulate five formations which are 9, 10, 12, 2, 3 o’clock 
direction from the line of sight of the test subject as portrayed in Fig. 4. Each camera 
record and save videos in remote storage (server) at the same time to yield the dataset D 
of machine learning. After finishing all experiments, we facilitated ML-kit algorithms 
to extract all possible coordinates among 33 landmarks. Since the camera yields 60 FPS 
(frame per second), we pullout 60 images per second. Finally, we secured 1,816,704 
images from the experiments. Using the images, we extract all landmarks. 
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Table 2. Anonymized test subjects 


No Age Height (cm) Gender No Age Height Gender 
1 21 156 Female 8 31 158 Female 
2 23 156 Female 9 23 170 Male 

3 23 162 Female 10 24 172 Male 

4 27 151 Female 11 26 170 Male 

5 23 173 Male 12 24 166 Female 
6 24 176 Male 13 25 180 Male 

7 23 166 Female 


ML-kit extracts 1,773,234 coordinates of landmarks after dropping NONE values 
which failed to get coordinates by ML-kit, whichis N = 1, 773, 234inD = {(0;, yoy E 
The x; is a vector with size R!*”4 consist of 3D coordinates (x, y, Z) from landmark #11, 
#12, #13, #14, #15, #16, #23, and #24 in Fig. 4. To acquire y; (label values, or equivalently 
ground truth values), we build mapping table for each action in Table 3. 


Table 3. Camera mapping to generate y; 


Landmark! 12 14 16 11 13 15 
Coordinate” x12 |y12 X14 | M4 | X16 | ¥16 |X | Yin [X13 |y13 | %I5 | 15 
M 1 0 0 0 0 0 0 0 0 0 0 0 0 
9 2 0 0 0 0 0 0 0 0 0 0 0 0 
: 3 4 4 4 4 4 4 1 1 1 1 1 1 
o 4 1 1 1 1 1 1 4 4 4 4 4 4 
n? 5 4 4 4 4 4 4 1 1 1 1 1 1 
6 1 1 1 1 1 1 4 4 4 4 4 4 
7 4 4 4 4 4 4 1 1 1 1 1 1 
8 4 4 4 4 4 4 1 1 1 1 1 1 
9 1 1 1 1 1 1 4 4 4 4 4 4 
10 (0) 0 0 0 0 0 0 0 0 0 0 0 
11 0 0 0 0 0 0 0 0 0 0 0 0 
12 1 1 1 1 1 1 4 4 4 4 4 4 
13 1 1 1 1 1 1 4 4 4 4 4 4 


Note: !landmark: index is referenced from Fig. 4, Z coordinate: subscript of x, y in coordinates 
is the index of landmark, and 3 motion: index is referenced from Table, all value in table is the 
camera index from Fig. 4 
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We set y; to be R!*!? since the final task of prediction is elbow angle of right and left 
in this paper. To compute right and left angle, we apply Eq. (1) landmark #12, #14, #16 
for right elbow and #11, #13, #15 for left elbow angle, respectively. We divided D into 
train and test set applying ration 80% and 20%, respectively. Finally, we trained train 
set by applying MLR, and predicted y; after data learning (i.e., after all train steps). 


4 Result Analysis 


4.1 Prediction Performance in RMSE 


We predict y; of test set (landmark #11, #12, #13, #14, #15, #16, #23 values) using x; of 
test set via MLR trained model. Let the predicted value be ĵ;. Then we compute RMSE 
errors between y; and ĵ; using Eq. (2). The overall error is 0.07064 and the least error is 
0.02726 shown in right wrist x-coordinate and the largest error is 0.11332 found in left 
elbow y-coordinate, respectively. The RMSE comparison is shown in Table 4 and visual 


representation is portrayed in Fig. 5. 


RMSE for each prediction criterion 


RMSE 


Ly 


left_wrist_y 


Tight_elbow_x 
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left elbow x 
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Fig. 5. Prediction RMSE errors between y; and ĵ;. 


4.2 Prediction Errors Between Left- and Right-Hand 


To compare the prediction behavior between right and left, we report scatter plots of 
shoulder and elbow in Fig. 6. We omit other plots due to the page limit. 


Prediction 


e 
in 


Prediction 


Enhancing the Measurement of the Range of Motion 


Table 4. Prediction RMSE errors between y; and ĵ;. 


Landmark index Description RMSE 
12 Right shoulder x 0.05932 
Right shoulder y 0.05330 
14 Right elbow x 0.09114 
Right elbow y 0.10460 
16 Right wrist x 0.02726 
Right wrist y 0.07429 
11 Left shoulder x 0.06468 
Left shoulder y 0.06330 
13 Left elbow x 0.09106 
Left elbow y 0.11332 
15 Left wrist x 0.03212 
Left wrist y 0.07331 
Overall Prediction Error 0.07064 


0.3 0.4 os 0.6 
Actual right shoulder x 


07 08 0.2 


Prediction 
o 
= 


T T T + T 
04 Os 0.6 0.7 0.8 


Actual right shoulder y 


0.2 0.4 06 
Actual right elbow x 


08 0.0 


04 06 0.8 10 
Actual right elbow y 


Fig. 6. Comparison prediction errors between left- and right-hand. 
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4.3 Performance of ROM Measurement 


Since elbow angles is only considered among ROM in this study, we averaged all RMSEs 
using observed coordinates and predicted coordinates via MLR, respectively. Note that 
the ground truth iS Yangle_from _groundTruth using y in Sect. 3 and we have two types of ĵ 
such that y Yangle_from_observe and y Yangle_form_MLR for angle comparison. The comparison 
between two angle estimation is reported in Table 5. 


Table 5. Prediction RMSE errors between elbow angle estimation approach. 


Computing Using coordinates Using coordinates Performance 
approach directly from camera (a) predicted via MLR (b) enhancement” 
Left elbow 129.02 117.26 9.1% 

Right elbow 111.35 96.44 13.4% 
Average 120.19 106.85 11.1% 


* Performance enhancement is estimated such that a—b x 100 


5 Conclusions 


It is an important task to estimate ROMs as accurate as possible in rehabilitation exer- 
cise. Since the computer vision technology is plausible to apply into detecting human 
landmarks, it is promising in motion analysis in that cost effective and time saving. How- 
ever, detecting and measuring the landmarks is not trivial yet. We addressed the way of 
measuring ROMs facilitating machine learning approach. In this paper, simple MLR is 
used to learn human elbow angle based on data. We developed how the machine learning 
approaches can be used and generate training and test dataset. Even though the test cases 
and test subjects are limited, we think the possibility of our approach could be addressed 
sufficiently. After implementation and experiments, we showed that machine learning 
based ROM estimation is possible approach in terms of enhancing the accuracy. If more 
enhanced algorithm is tested and evaluated in our domain, the better enhancement will 
be achieved. We leave this as our future research. 
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Abstract. Frailty is a clinical syndrome associated with ageing that 
characterizes an intermediate state between robust health and loss of 
autonomy. To preserve the abilities of older adults and prevent depen- 
dency, it is important to identify and evaluate their frailty. This approach 
is part of a dependency prevention strategy, based on a thorough under- 
standing of their medical, social, and living environment. This under- 
standing is usually acquired through significant data collection using 
standardized evaluation surveys. The obtained data is then analyzed 
to provide personalized recommendations for the beneficiaries’ lifestyles. 
Our article presents the concept of frailty and a personalized recommen- 
dation system aimed at helping citizens prevent frailty. This system uses 
an innovative self-assessment approach designed for older adults, without 
necessarily involving healthcare professionals. 


Keywords: Recommendation systems - Elderly - Frailty - 
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1 Introduction 


As the global population ages, preserving physical, cognitive, and psycho-social 
abilities is essential for healthy aging. Age-related declines in these abilities can 
lead to frailty, a syndrome characterized by a decline in physiological systems, a 
decrease in functional reserves, and vulnerability to stress [4]. Unlike disability, 
frailty can often be reversed or minimized with personalized interventions [16] 
aimed at preserving abilities. 
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Managing frailty involves a three-step process, including frailty screening and 
the definition of a personalized intervention plan, as well as follow-up to prevent 
or delay disability and dependence and improve the quality of life of elderly 
individuals [8]. Several European countries, including France, are developing 
innovative solutions for elderly individuals and their caregivers to improve their 
quality of life in the context of the Silver Economy [5]. Our work is aligned 
with this goal of promoting « Successful Ageing » and behaviours that promote 
good health by creating a personalized service recommendation engine for the 
prevention of frailty. 

In this article, we provide an overview of the medical context and issues 
related to population ageing that motivated our research. We also discuss the 
link between frailty and intrinsic capacities. In the third section, we present our 
innovative recommendation platform, « Senselife », which is based on a thorough 
analysis of user profiles and provides personalized recommendations for services 
to meet their specific needs. Finally, we conclude the article by summarizing our 
results and discussing future perspectives for practical applications to improve 
the quality of life of elderly individuals. 


2 Context 


In order to comprehend the driving forces behind our research and the obstacles 
to be surmounted in advancing a recommendation engine, it is necessary to 
acquire a foundational understanding of the medical knowledge surrounding the 
factors that contribute to loss of autonomy in older adults, as well as how this 
can be addressed in an ongoing process of care. 


2.1 Frailty and Intrinsic Capacities 


The syndrome of frailty is characterized by a decline in physiological reserve 
capacities that impact adaptation mechanisms to stress. Psychological, social, 
economic, and behavioural factors, as well as comorbidities, influence its clinical 
manifestation. It is linked to increased mortality risk, disability, falls, hospitaliza- 
tion, and institutionalization [12]. Consequently, early interventions to prevent 
the progression of frailty and disability are crucial. Frailty evaluation allows older 
adults to be classified into three categories: robust, pre-frail, and frail. 

Prevalence of frailty and pre-frailty in EU countries is around 15% and 50%, 
respectively, among those aged 65 and older based on the study [13]. 

The concept of intrinsic capacities (IC) was introduced by the World Health 
Organization (WHO) to enable a more comprehensive and suitable evaluation of 
the ageing population. IC refers to the sum of all physical and mental abilities an 
individual can use throughout their life and includes five domains that summarize 
their capacities [1]. 

Understanding the relationship between IC and frailty is crucial. Indeed, 
frailty is a state of vulnerability that can occur in elderly people due to a decrease 
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in intrinsic capacities. Thus, intrinsic capacities are the basic capacities that an 
elderly person needs to function independently. 

Detecting and evaluating frailty is challenging due to the variety of available 
measurement tools. A review by [6] identified 29 different measures, ranging 
from simple screening instruments to complex ones. Meanwhile, the assessment 
of intrinsic capacities (IC) in older adults is a growing field. A proposed scoring 
system by [9] assesses IC in older adults and highlights the complementarity 
between measuring frailty and IC. Conducting a comprehensive evaluation of 
both frailty and IC is crucial for enhancing the quality of life and well-being of 
older adults. 


2.2 Frailty Personalized Intervention Plans Implementation 
Challenges 


To manage frailty in older adults, a personalized intervention plan involving 
collaboration between multiple stakeholders is necessary (up to 13 actors can 
be included) [2,15]. However, its implementation can be complex and resource- 
intensive, limiting accessibility to a significant number of older adults. 

Creating a personalized intervention plan for frailty in older adults requires 
a comprehensive evaluation of the individual’s situation, including measures of 
intrinsic capacity and frailty. Identifying available services and personal infor- 
mation, such as preferences and history, also play a crucial role in developing an 
effective plan. 

To develop an accurate and dynamic personalized intervention plan, a deci- 
sion support system capable of recommending services taking into account their 
medical, social, and environmental profile would be useful. To this end, we will 
present our recommendation platform, called Senselife, in detail in the following 
section. It will enable an interactive, precise, and personalized response to the 
specific needs of each older adult, based on their frailty situation and personal 
preferences. 


3 Seneselife Platform 


The goal of the Senselife platform is to address the challenge of managing frailty 
in older adults by utilizing self-evaluations to provide personalized recommen- 
dations tailored to their needs and preferences. The platform offers a variety 
of features such as the ability for users to express their living conditions and 
match supply and demand for services. Additionally, it provides a selection of 
services aimed at strengthening the functional abilities of older adults based on 
their intrinsic capacity. However, the platform needs to define and use knowledge 
to create tailored recommendations. This requires identifying key knowledge to 
characterize users and services, representing it, and integrating it with the soft- 
ware structure to improve recommendations. The scheme of the platform is illus- 
trated in Fig. 1, which is composed of three main components: data collection, 
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Fig. 1. Senselife Platform 


recommendation generation, and services consumption. Citizen satisfaction and 
service utilization are used to adapt future recommendations. 

The platform’s recommendations will heavily rely on the data it collects. The 
primary sources of this data will be the citizen’s self-evaluation survey responses 
and connected devices. The survey’s questions are chosen from different frailty 
surveys. They were selected based on whether the citizen can answer them alone 
(for example, ADL! questions) and whether they provide an understanding of 
the user’s condition. Additionally, service providers are responsible for defining 
their services on the platform, which is the second input for the recommendation 
engine. 

The collected data will feed the recommendation engine, which will present 
the citizen with a list of services that meet their needs, factoring in their personal 
characteristics and preferences. 


3.1 Citizen Profile 


In the context of recommendation, user profiling is the process of collecting 
and analyzing information about users to understand their preferences, habits, 
and behaviors. It is a key element in improving the relevance and quality of 
recommendations [7]. To this end, we have developed a conceptual model of 
the user profile for our platform, which is the elderly citizen in our case. This 


1 ADL: Activities of Daily Living survey. 
https://www.alz.org/careplanning/downloads/katz-adl.pdf. 
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model takes into account all relevant elements to characterize the citizen while 
remaining extensible. As shown in Fig. 2a, a citizen profile consists of 5 parts: 
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Fig. 2. Excerpt from the Senselife meta-model 


— Personal Information: This section contains the citizen’s personal data, 
such as age, name, phone number, address, etc. 

— Goal: This section describes the person’s overall goal. This information helps 
the recommendation engine to better understand the person’s motivations 
and establish a link with their intrinsic capacities. For example, the goal of 
“establishing new friendships” can be associated with the category “Social 
relationships” which is linked to psychological capacity. 

— Activity Preferences: This section provides more details on the person’s 
activity preferences and their respective category. 

— Recommendation Preferences: Helps the recommendation engine to filter 
the recommended services based on the person’s preferences, for example, by 
specifying a preference for well-rated services. 

— Personal Characteristics: Are obtained from responses to the survey and 
can be classified into different categories of IC. For example, the character- 
istic “Vision problem” can be identified from the “Comprehensive Geriatric 
Assessment” survey [11] and is associated to the visual capacity [10]. 


Although our platform can generate an estimated frailty score based on the 
data collected, it’s essential to acknowledge that this score may not be as accurate 
as those obtained by healthcare professionals. For example, the 6 questions of 
the ADL survey can help us to understand a specific aspect of the person and 
have a frailty score on a scale of 6. Other scores are also calculated which will 
give us more comprehension about the person. In summary, our primary goal is 
to gather information about the citizen to better understand their situation and 
develop personalized recommendations, rather than providing an exact frailty 
score. 
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3.2 Services 


Service providers are responsible for creating services on the platform. We offer 
a service model that will assist providers in creating their services while enabling 
the recommendation engine to efficiently comprehend the input data. Figure 2b 
displays this model. 


— Service attributes: General information about the service. This information 
is crucial for grouping available services. It can also be used to filter options 
and help elderly people make their choices. 

— Targeted Beneficiaries: The group of citizens targeted by this service. It 
is important to note that some services may be limited to specific profiles. 

— Functionality: There are various interventions, such as multimodal exer- 
cises, that contribute to multiple domains of IC. Therefore, when defining 
the service, the provider can specify the domains affected by this service. 

— Category: Services can take various forms, whether as autonomous activities 
performed by the individual, products provided, or services provided by other 
individuals or organizations. This will allow for more effective identification 
of services that meet the specific needs of the elderly person. 

— Service quality evaluation: Service quality is crucial for recommendations. 
Evaluations help assess quality, allowing beneficiaries to judge and future 
beneficiaries to gain insight. 


3.3 Recommender Engine 


The Senselife platform employs a recommendation engine that provides services 
for frailty prevention. With the help of algorithms, this recommendation engine 
suggests specific services to elderly individuals based on their personal informa- 
tion and circumstances. The engine evaluates the data provided by the elderly 
person and suggests the most relevant services using the knowledge bases and the 
rule base available. The knowledge base contains the different personal charac- 
teristics, needs, the service types and the relation between them. The rule base 
contains the rules that will be used to manipulate the data in the knowledge 
base. The validation of this rules are based on the expertise of the health-care 
professionals. 

There are various recommendation techniques, such as collaborative filter- 
ing algorithms, content-based algorithms, and knowledge-based algorithms. The 
choice of algorithm depends on the specific requirements and objectives of the 
recommendation system [14]. Our recommendation engine uses a knowledge- 
based recommendation technique that enables increased personalization. This 
technique analyze information about users, items, or their relationships to sug- 
gest relevant items using a knowledge base. This approach infers the relationship 
between a user’s needs and a possible recommendation, as explained in [3]. 
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Fig. 3. Recommendation Engine overview 


When we zoom-in on Fig. 1 in the recommendation engine section, we can 
see Fig.3, which provides a schematic representation of the knowledge-based 
recommendation technique. Ultimately, the elderly person receives a filtered rec- 
ommendation based on their preferences. 


4 Conclusion and Perspectives 


The article focuses on the vulnerability of elderly people and suggests person- 
alized service recommendations for frailty prevention. The proposal is a recom- 
mendation platform that assesses the older person in a multidimensional way, 
taking into account both intrinsic abilities and measures of frailty. 

This approach is crucial in understanding an individual’s needs to determine 
the most suitable services. The platform uses a self-evaluation approach, which 
is user-friendly and accessible to everyone. The recommendation technique of the 
engine is presented (knowledge-based one), with an explanation of its different 
steps and the knowledge bases used. 

The platform provides a holistic solution to ensure that elderly people receive 
the necessary support without relying solely on healthcare professionals. To 
achieve this goal, the approach requires attention to various aspects. We identify 
several research avenues for future work, including validating citizen and service 
profile models, verifying the effectiveness of the incremental approach, and con- 
sidering the optimization and complexity aspects of the algorithms used by the 
recommendation engine. 
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Abstract. This study systematically reviewed research that applied dual-task 
interventions using VR technology for balance and cognitive tasks among older 
adults. Ten databases were searched following the PRISMA guideline, and 18 
studies were selected based on their evidence levels and risk of bias. The selected 
studies consisted of 10 RCTs, 6 non-randomized controlled trials, and one each 
of case and qualitative studies. The types of balance tasks included standing and 
sitting postures, and all studies utilized cognitive tasks that required concentration. 
A total of 30 physical assessment tools and 42 cognitive assessment tools were 
identified. The results showed that virtual reality interventions improved balance 
and cognitive abilities among older adults and had a positive effect on fall pre- 
vention. These findings suggest that VR technology can be an effective tool for 
improving the physical and cognitive health of older adults. 


Keywords: Virtual reality - Dual-task - Cognition - Balance - Elderly 


1 Introduction 


VR is being used by integrating various technologies to experience realistic represen- 
tations in a virtual space. Research on dual-task training using VR has been conducted 
on a diverse range of subjects, including individuals with Parkinson’s disease, as well as 
children, adults, and the elderly of all genders and ages. While numerous interventions 
involving VR programs and devices for dual-task training have been conducted in for- 
eign countries, research utilizing these technologies for such training is still lacking in 
Korea. 

Dual-task training involves performing cognitive tasks while engaging in motor 
activities, creating a situation of cognitive-motor interference where efforts are divided 
between two tasks [1]. This induces a state of conflict, requiring high levels of attention 
and concentration during the execution of motor tasks. 
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Dual-task training is easily implemented through VR technology, allowing for per- 
sonalized exercise, rehabilitation, and interventions for individuals. This approach can 
be particularly beneficial for older adults and patients requiring rehabilitation, as it can 
stimulate interest and promote engagement, leading to enhanced daily functioning and 
recovery [2—4]. 

It should be noted, however, that the application of VR technology also has its 
disadvantages, with cybersickness being identified as a common issue among some 
users [5]. This is a particularly critical issue for individuals with disabilities and the 
elderly [6]. Head-mounted displays (HMDs) have been identified as a major cause of 
cybersickness [7, 8]. Therefore, for the elderly, VR programs utilizing technologies other 
than HMDs are more suitable. 

This study aims to systematically review domestic and international research on dual 
tasks that combine balance and cognitive tasks using VR for older adults, examining 
the level of evidence, types of tasks used, validation tools, intervention duration, and 
effects. It intends to provide directions for VR-based dual task research in Korea and 
offer evidence-based approaches for researchers working with older adults. 


2 Methods 


2.1 Literature Selection and Study Design 


This study examined the effects of dual tasks combining balance and cognitive tasks 
using VR devices on older adults. The literature selection process was conducted using 
the Preferred Reporting Items for Systematic Reviews (PRISMA) guidelines. The level 
of evidence of the selected studies was evaluated using the evidence level assessment 
developed by Abresman, Scheer, and Lieberman (2008) and divided into five levels [9]. 


2.2 Database and Search Terms 


To find relevant studies for the topic, a three-step search strategy was employed between 
October and December 2022. Firstly, previous research trends were identified using med- 
ical and health literature analysis databases (PubMed, CINAHL) and Google Scholar. 
Secondly, search terms were selected. Lastly, systematic reviews that had been pre- 
viously conducted on topics related to VR were consulted to select the paper search 
databases [10, 11]. 

For domestic databases, the search was conducted using the following search terms: 
“Elderly” AND “Virtual reality” AND “Balance” AND “Dual task” AND “Cognition” on 
the Research Information Sharing Service (RISS). For foreign databases, the search was 
conducted using the same search terms on ACM Digital Library, SCOPUS, PubMed, 
ScienceDirect, CINAHL, EMBASE, Web of Science, PsycINFO, and IEEE Xplore. 
Studies conducted between 1960 and December 2022 were included. After collecting 
all papers retrieved from each database, the final papers were selected according to the 
following selection and exclusion criteria. In order to ensure that the selected studies 
were relevant to the research topic, a set of selection criteria and exclusion criteria were 
used. The inclusion criteria were: (1) Studies that targeted older adults, (2) Studies that 
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used virtual reality to mediate dual tasks related to balance and cognitive tasks, (3) Papers 
written in Korean or English, and (4) Studies for which the full text is available. The 
exclusion criteria were: (1) Studies that did not use VR technology as an intervention, (2) 
Studies that used head-mounted display (HMD) devices for mediation, (3) Studies that 
targeted older adult patients with central nervous system disorders or chronic diseases, 
(4) Systematic reviews and meta-analyses, and (5) Thesis studies and protocol studies. 


2.3 Method of Results Analysis 


For the selected studies, we analyzed the types of dual tasks combining balance and 
cognitive tasks that elderly individuals perform using VR devices. First, we divided 
the types of balance and cognitive tasks performed during dual-task interventions using 
VR and compiled them into a table. Second, we analyzed the assessment tools used to 
evaluate the effectiveness of dual-task interventions using VR and compiled the results 
into a table. 


3 Results 


3.1 Selection of Studies 


A total of 3983 studies were retrieved from the initial search of 10 databases. After 
removing duplicates (n = 421) using EndNote and manual review, we assessed the titles 
and abstracts of the remaining papers for relevance to Virtual Reality, dual-task, and 
balance, and excluded irrelevant papers, such as review and meta-analysis papers, papers 
without full texts, papers with inappropriate study designs, and papers with participants 
not relevant to the research topic. After the first screening, we collected 137 studies 
and further excluded irrelevant papers by checking the full texts, resulting in the final 
selection of 18 papers (Fig. 1). 


3.2 Level of Evidence 


The evaluation of the level of evidence for the final selection of 18 studies revealed that 
10 studies were classified as level 1, accounting for 55.6% of the total. Three studies 
each were classified as level 2 and level 3, accounting for 16.7% each. Additionally, 
there were two studies (11.0% in total) classified as level 5 in the evidence level, which 
included one case report and one qualitative study (Table 1). 


3.3 Types of Balance and Cognitive Tasks in Dual-Task Interventions 


The analysis of the dual-task intervention using VR, divided into balance task types 
and cognitive task types, showed that the balance task type had the highest number of 
tasks in the standing position, with 15 tasks (83.3%). Two tasks (11.1%) were performed 
in the seated position, and one task (5.6%) included both standing and seated balance 
tasks. For the cognitive tasks, attention had the highest number of tasks with 18 (32.1%), 
followed by memory with 16 (28.6%), and executive function with 14 (25.0%). Planning 
tasks accounted for 4 (7.1%), and calculation and inhibition/interference control each 
accounted for 2 (3.6%) (Table 2). 
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Fig. 1. Flow Diagram of the Literature Selection Process 


Table 1. Level of Quality of Each Study 


Evidence level Definition Frequency, n (%) 
1 Randomized controlled trials 10 (55.6) 
2 Two groups non-randomized studies 3 (16.7) 
3 One groups non-randomized studies 3 (16.7) 
4 Single subject designs 0 (0.0) 
Surveys 
5 Case reports 2 (11.0) 
Narrative literature reviews 
Qualitative researches 
Total 18 (100.0) 
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Table 2. Classification of Balance Types and Cognitive Types of Double Tasks 


Dual task Intervention n(%) Total 
Types Details 
Balance types Standing Balance 15 (83.3) 18 (100.0) 
(Static Balance, Dynamic Balance) 
Sitting Balance 2 (11.1) 
(Static Balance, Dynamic Balance) 
Standing, Sitting Balance 1 (5.6) 
(Static Balance, Dynamic Balance) 
Cognition types Attention 18 (32.1) 56 (100.0) 
Memory 16 (28.6) 
Executive Function 14 (25.0) 
Planning 4 (7.1) 
Calculation 2 (3.6) 
Inhibition/Interference Control 2 (3.6) 


3.4 Analysis of Evaluation Tools for Assessing the Effects of VR-Based Dual-Task 
Interventions 


Through the selected studies, a total of 30 evaluation tools were identified for measuring 
the effects of VR-based dual-task interventions on physical functions of older adults, 
including 26 for balance assessment and 4 for motor function evaluation. Among the 
balance assessment tools, Mini-BESTest and TUG were the most frequently used, with 
3 studies (10.0%) each. Motor function evaluation was conducted with 4 tools, including 
fNIRS. A total of 42 evaluation tools were used for cognitive function assessment, which 
was further categorized into executive function, memory, attention, MCI screening test, 
and overall cognitive function assessment tools. TMT was the most commonly used 
tool (8 studies, 19.0%) for executive function assessment, while WMS-R was the most 
frequently used tool (3 studies, 7.1%) for memory assessment. All 5 studies (11.9%) 
assessing attention utilized Stroop. MoCA was the most frequently used tool (6 studies, 
14.3%) for MCI screening. For overall cognitive function assessment, fNIRS and VFT 
were the most commonly used tools, each in 2 studies (4.8%) (Table 3). 


Table 3. Measured Dependent Variables and Outcomes 
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Dependent variable Outcomes n (%) Total 
Types Details 
Physical | Balance Mini-BESTest 3 (10.0) | 26 (86.7) 
= 20), |(@=26) TUG 3 (10.0) 
FGA 2 (6.7) 
FSST 2 (6.7) 
SFT 2 (6.7) 
SPPB 2 (6.7) 
10MWT 2 (6.7) 
BBS 1 (3.3) 
BlIORescue 1 (3.3) 
EEG 1 (3.3) 
FES-I 1 (3.3) 
Five Times Sit to Stand test | 1 (3.3) 
GUG 1 (3.3) 
Incident Rate of Falls 1 (3.3) 
LoS 1 (3.3) 
The Community Balance and | 1 (3.3) 
Mobility Scale 
YBT 1 (3.3) 
Motor Function fNIRS 1 (3.3) | 4 (13.3) 
(n= 4) ten-point Borg scale 1 (3.3) 
Wasps hit 1 (3.3) 
30 s Sit-to-Stand Test 1 (3.3) 
Cognition | Executive Function TMT-A, TMT-B 8 (19.0) | 11 (26.2) 
m=42) |m=11) 
VST 2 (4.8) 
EXIT-25 1 (2.4) 
Memory (n = 7) WMS-R 3 (7.1) |7 (16.7) 
ADAS 1 (2.4) 
CCVLT 1 (2.4) 
DST 1 (2.4) 


(continued) 
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Table 3. (continued) 


Dependent variable Outcomes n (%) Total 
Types Details 
The spatial n-back task test 1 (2.4) 
Attention (n = 5) Stroop 2 (4.8) |5 (11.9) 
CWT 1 (2.4) 
Simple stroop word color test | 1 (2.4) 
TOVA 1 (2.4) 
Screened as MCI MoCA 6 (14.3) | 8 (19.0) 
(n= 8) K-MMSE 1 (2.4) 
MMSE 1 (2.4) 
Overall Cognitive Function (n | {NIRS 2 (4.8) | 11 (26.2) 
= 1) VFT 2 (4.8) 
CNT40 1 (2.4) 
DSST 1 (2.4) 
EEG 1 (2.4) 
RBANS 1 (2.4) 
Self-Reported Cognitive 1 (2.4) 
Function 
WAIS-R 1 (2.4) 
WAIS-4 1 (2.4) 


ADAS: Alzheimer’s Disease Assessment Scale, BBS: Berg Balance Scale, CCVLT: The Chinese 
version of the California Verbal Learning Test, CNT40: Computerized nerocognitive function test, 
CWT: The Chinese version of the Stroop Colour and Word Test, DSST: Digit Symbol Substitution 
Task, DST: Digit Span Test, EEG: Electroencephalogram, EXIT-25: The Executive Interview 25, 
FES-I: The Falls efficacy scale, FGA: Fuctional Gait Assessment, fNIRS: Functional NIRS, FSST: 
Four Square Step Test, GUG: Get-up-and-go Test, HbO2: Oxygenated Hemoglobin, LoS: Limits 
of Stability Scores, MBT: Mind-Body Trainer, K-MMSE: Korea-Mini Mental State Exam, Mini 
BESTest: Mini Balance Evaluation Systems Test, MMSE: Mini Mental State Exam, MoCA: 
Montreal Cognitive Assessment, RBANS: the Repeatable Battery for the Neuropsychological 
Status, SFT: Senior Fitness Test, SPPB: Short Physical Performance Battery, TMT-A: Trail making 
test-A, TMT-B: Trail making test-B, TOVA: Test of Variables of Attention, TUG: Timed Up and 
Go, VFT: Verbal Fluency Test, VST: Victoria Stroop Test, WAIS-R: Wechsler Adult Intelligence 
Scale-Revised, WAIS-4: Wechsler Adult Intelligence Scale-4, WMS-R: Wechsler Memory Scale- 
Revised, YBT: Y-Balance Test, LOMWT: 10m Walking Test. 


4 Conclusion 


In this study, we examined research that provided dual-task interventions combining 
balance and cognitive tasks using VR. We analyzed the types of balance and cognitive 
tasks used to provide dual-task performance through VR, evaluation tools for analyzing 
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the effects of dual-task performance using VR on the elderly, and the physical, cognitive, 
and emotional effects, fall prevention rates, and usability of dual-task performance with 
VR technology. VR dual-task interventions for elderly individuals may have positive 
effects on their physical balance and cognition. However, it is important to select appro- 
priate challenging tasks based on their cognitive abilities and to create diverse content 
that can stimulate their interest. Furthermore, it has been observed that while there are 
numerous existing assessment tools for evaluating physical and cognitive performance 
separately in the literature review of previous studies, there is a lack of assessment tools 
specifically designed to evaluate dual-task performance itself. VR technology facilitates 
the integration of physical and cognitive tasks in dual-task paradigms. Therefore, there 
is a need for research on the reliability and validity of assessment tools that can evaluate 
dual-task performance using VR technology. 


References 


1. Woollacott, M., Shumway-Cook, A.: Attention and the control of posture and gait: a review 
of an emerging area of research. Gait Posture 16(1), 1-14 (2002) 

2. Hwang, J.-H., Park, M.-S.: Effect of a dual-task virtual reality program for seniors with mild 
cognitive impairment. Korean J. Clin. Lab. Sci. 50(4), 492-500 (2018) 

3. Pedroli, E., et al.: Characteristics, usability, and users experience of a system combining 
cognitive and physical therapy in a virtual environment: positive bike. Sensors 18(7), 2343 
(2018) 

4. Thalmann, M., et al.: Usability study of a multicomponent exergame training for older adults 
with mobility limitations. Int. J. Environ. Res. Public Health 18(24), 13422 (2021) 

5. Byun, H., Park, C.W.: Introduction of cybersickness. Korean J. Otorhinolaryngol. Head Neck 
Surg. 62(10), 545-553 (2019) 

6. Tuena, C., et al.: Usability issues of clinical and research applications of virtual reality in 
older people: a systematic review. Front. Hum. Neurosci. 14, 93 (2020) 

7. Buhler, H., Misztal, S., Schild, J.: Reducing VR sickness through peripheral visual effects. 
In: 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE (2018) 

8. Chang, E., et al.: An integrated model of cybersickness: understanding user’s discomfort in 
virtual reality. J. KIISE 45(3), 251-279 (2018) 

9. Moher, D., et al.: Preferred reporting items for systematic reviews and meta-analyses: the 
PRISMA statement. Ann. Intern. Med. 151(4), 264-269 (2009) 

10. Bevilacqua, R., et al.: Non-immersive virtual reality for rehabilitation of the older people: a 
systematic review into efficacy and effectiveness. J. Clin. Med. 8(11), 1882 (2019) 

11. Dermody, G., et al.: The role of virtual reality in improving health outcomes for community- 
dwelling older adults: systematic review. J. Med. Internet Res. 22(6), e17331 (2020) 


312 J.-S. Choi et al. 


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 
International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, 
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate 
credit to the original author(s) and the source, provide a link to the Creative Commons license and 
indicate if changes were made. 

The images or other third party material in this chapter are included in the chapter’s Creative 
Commons license, unless indicated otherwise in a credit line to the material. If material is not 
included in the chapter’s Creative Commons license and your intended use is not permitted by 
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from 
the copyright holder. 


® 


Check for 
updates 


Comparative Usability Testing Between 
Lightweight Power Wheelchairs: Focused 
on People with Physical Disabilities 
in the Community 


Yeong-hun Kwon!, Dong-wan Kim!, and Jongbae Kim? ©) 
1 Department of Occupational Therapy, The Graduate School, Yonsei University, Seoul, 
South Korea 
Yhkwon97@yonsei.ac.kr 
2 Department of Occupational Therapy, College of S/W Digital Healthcare Convergence, 
Yonsei University, Wonju, South Korea 
jongbae@yonsei.ac.kr 


Abstract. There have been studies on Lightweight Power Wheelchairs (LPW) 
that compensate for the disadvantages of large and heavy powered wheelchairs for 
the activities of the people with physical disabilities in the community, but there is a 
lack of research to understand usability. Accordingly, we compared the usability of 
LPWs developed in Korea for 5 wheelchair participants and caregivers to find out 
their effectiveness, efficiency, and satisfaction. As a result of the usability testing, 
there was a difference between the two LPWs in effectiveness and satisfaction. 
LPW1 was analyzed as grade B/’Good’ usability LPW2 was analyzed as grade 
ABest Imaginable’ usability. The usability of the heavier LPW2 was highly 
appreciated. This can be interpreted that users feel a sense of stability in the LPW 
with a certain amount of weight. In future studies, it is considered necessary to 
study the appropriate weight of LPW that users can feel stable. 


Keywords: Lightweight Power Wheelchair - Usability Testing - People with 
Disabilities 


1 Introduction 


According to the Ministry of Health and Welfare of Korea in 2020, the rate of physical 
disability among community-dwelling people with disabilities were the highest [1]. 
People with severe physical disabilities use mobility assistive devices when moving to 
the community. Among them, powered wheelchairs are generally used when users cannot 
propel independent wheelchairs, and according to Lee’s study, powered wheelchair users 
reported low satisfaction with the weight and size of powered wheelchairs and post- 
service [2]. Accordingly, Lightweight Power Wheelchairs (LPW) with the function of 
powered wheelchairs at home and abroad are being studied [3-6]. 

Usability is specified according to the ISO-9241-11 guidelines, and usability testing 
is to investigate the interaction between users and products in the process of constructing 
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and executing usability scenarios [7]. According to Kim, in order to increase the usability 
of assistive devices, it is important to reflect the needs of the people of disabilities from 
the early stage of development, and to this end, usability testing is required [8]. However, 
research on usability testing of LPW is insufficient at domestic and abroad. 

Kim conducted a comparative usability testing on a hybrid manual-powered 
wheelchair but focused on wheelchair driving [9]. In the case of people with severe phys- 
ical disabilities, it is difficult to manage independent powered wheelchairs, so help from 
caregivers is essential. Accordingly, it is important to find out the usability of wheelchairs 
from the perspective of caregivers in product development and improvement [10]. 

Therefore, in this study, we would like to compare the usability of two LPWs recently 
developed in Korea in accordance with the ISO-9241-11 guidelines from the perspective 
of users and caregivers. Through the analysis of the usability testing results, we intend to 
derive factors to be considered in order to increase the usability of users and caregivers 
and to provide basic data for future LPW research. 


2 Method 


2.1 Participants 


Participants in this study were recruited from people with physical disabilities who 
use powered wheelchairs. According to Nelson’s research, 5 people with physical dis- 
abilities who agreed to participate in the study were finally selected [11]. The general 
characteristics of the study participants are shown in Table 1. 


Table 1. Characteristics of participants 


Characteristics Participant 
P1 P2 P3 P4 P5 
Gender Male Male Male Male Male 
Age 52 58 45 68 65 
Diagnosis SCI SCI SCI Polio SCI 
Type of wheelchair Power Manual/ Power Manual/ Power 
Power Power 


2.2 Inclusion and Exclusion Criteria 


If the participants met (1) A power wheelchair can be used independently and (2) 
There is no cognitive difficulty in understanding the research process and agrees to 
the research voluntarily, they were included in the study, and if they met (3) Do not 
currently use a powered wheelchair and (4) Participant currently has difficulty using an 
power wheelchair, they were excluded from the study. 
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2.3 Tools 


2.3.1 Lightweight Power Wheelchair 


In this study, two types of LPW manufactured in Korea were used (Fig. 1). Unlike 
existing power wheelchairs, it is light in weight and can be folded and stored, so it can 
be carried in a vehicle. Another advantage is that the battery can be easily installed and 
removed. It is equipped with the same driving device as the existing power wheelchair 
and can be steered and speed controlled using a controller such as a joystick. 

Both LPWs have the same 8-inch front wheel and 14-in. rear wheel. The battery 
capacity of the two LPWs is the same at 24 V and 16 Ah. The motor specifications of 
the two LPWs are the same at 250 W and 4600 rpm. The controller of both LPWs is the 
same. The weight of the two LPWs is different. The weight of LPW1 is 29.7 kg, and 
the weight of LPW2 is 31 kg. The frame sizes of the two LPWs are different. LPW1 is 
105-58-84 cm (length, width, height), LPW2 is 103-58-96 cm (length, width, height). 


Lightweight Power Wheelchair 1(LPW1) Lightweight Power Wheelchair 2(LPW2) 


Fig. 1. Lightweight power wheelchair 


2.4 Usability Testing 


In the International Organization for Standardization (ISO), usability is defined accord- 
ing to the ISO-9241-11 standard. Usability refers to effectiveness, efficiency, and satis- 
faction when a user uses a system, device, or service to achieve a specific purpose [12]. 
Effectiveness is to measure whether the user has performed the desired task accurately 
and perfectly. Efficiency includes time, costs invested to achieve goals, the degree of 
effort, and fatigue. Satisfaction is the evaluation of an individual’s cognitive, emotional, 
and psychological responses, such as comfort and positive attitude, when a user per- 
forms a task using a product [12, 13]. Effectiveness was measured through task success. 
Efficiency measured (1) task performance time (TPT), (2) effort and fatigue required 
to perform tasks using the product through The Borg Rating of Perceived Examination 
Scale (RPE), and (3) subjective difficulty when performing tasks through the Differ- 
ential Rating Scale (DRS). Satisfaction measured acceptance, satisfaction, and overall 
usability of the product through (1) Acceptability Rating Scale (ARS) and (2) System 
Usability Scale (SUS) [14-17]. RPE is measured in the range of 6 to 20 points, 6 points 
are no exertion, 20 points are interpreted as maximum exertion, DRS is interpreted as 
—3 points to 3 points, —3 points to very difficult and 3 points to very easy. ARS ranges 
from —3 to 3 points, and —3 points are interpreted as very unacceptable and 3 points 
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as very acceptable. The SUS converts the participants’ responses into 100 points and 
substitutes them for the indicator to confirm satisfaction and overall usability (Fig. 2). 
For usability testing, 3 scenarios were created, and 18 detailed tasks were included 
under them. Each scenario is divided into a pre-driving process (S1), a driving process 
(S2), and a post-driving process (S3) (Table 2). S1 and S3 were performed by the care- 
givers, and S2 was performed by the participants. Usability testing was conducted by 
establishing a testbed similar to the environment that can be experienced when moving 
in a wheelchair in the community. The testbed was aimed at testing the stages of the 


driving process (Fig. 3). 


Table 2. Scenario for usability testing 


Scenario Task number Task 
S1 S1-Task 1 Connecting the developed 4-part cushion 
[Setting] S1-Task 2 Connecting a battery 
S2 S2-Task 1 Turn on the wheelchair, adjust the speed 
[Driving] S2-Task 2 Driving on a sidewalk block 

S2-Task 3 Driving on a 1/8 slope way 

S2-Task 4 Driving on a bumpy road 

S2-Task 5 Driving on a uneven road 

S2-Task 6 Driving on a S type road 

S2-Task 7 Driving on a drain 

S2-Task 8 Driving on a steep slope way 

S2-Task 9 Driving on a hollow road 

S2-Task 10 Driving on a speed hump 

S2-Task 11 Driving on a gravel road 

S2-Task 12 Driving through the doorway 

S2-Task 13 Driving on a braille block 

S2-Task 14 Driving on a side slope way 

S2-Task 15 Turn off the wheelchair at the arrival point 
S3 S3-Task 1 Disconnecting the battery 


[Finishing] 
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Detractor Passive 


J see NPS: 


Acceptable: 


Worst imaginabie oor OK Good Ere: 
menve anemian : 


) =] Grade: F D c B A 


SUS Score: 0 10 20 30 40 50 60 70 80 s0 100 


Fig. 2. Indicators of usability testing (RPE, DRS, ARS, SUS) 


Fig. 3. Wheelchair driving testbed 


2.5 Data Analysis 


Analysis was conducted to evaluate the effectiveness, efficiency, and satisfaction of 
each detailed task performed using LPW1 and LPW2. Effectiveness was expressed as a 
percentage by calculating the successful proportion of tasks performed by each LPWs. To 
measure efficiency and satisfaction, average values and standard deviations of TPT, RPE, 
DRS, ARS, and SUS of each LPWs were calculated using Excel and then substituted 
into each evaluation index. 


3 Result 


Section 3.1 was written based on the results of scenario 2 to find out about the usability 
of users using LPWs. Section 3.2 was written based on the results of S1 and S3 to find 
out the usability in setting the LPWs by the caregivers. 


3.1 S2 Tasks Performed by the Participants 


3.1.1 Effectiveness 


The effectiveness of the two LPWs was confirmed by analyzing the success or failure of 
the task. All participants succeeded in performing LPW1 except for one participant who 
did not succeed in performing T10. LPW2 was successfully performed by all participants 
at once in all scenarios. Thus, LPW1 showed 80% effectiveness and LPW2 showed 100% 
effectiveness. 
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3.1.2 Efficiency 


TPT were averaged 11.5 s (LPW1) and averaged 6.2 s (LPW2). RPE scores averaged 
9.7 points for LPW1 and 9.4 points for LPW2. Both LPWs were usable with very light 
effort. DRS scores averaged 1.3 points for LPW1 and 1.4 points for LPW2. Both LPWs 
were analyzed close to a barely easy level. (Table 3). 


Table 3. Result of S2 TPT, RPE, DRS 


Task | TPT (Aver. / SD) RPE (Aver. / SD) DRS (Aver. / SD) 
LPW1 LPW2 LPW1 LPW2 LPW1 LPW2 
1 10.2 (6.4)  |8.4(5.5) | 8.0(3.5) | 6.8 (1.6) 2.0 (1.5) 2.0 (2.0) 
2 6.4 (3.9) |5.0(2.5) | 8.4(2.3) | 88(3.1) 2.2 (0.7) 2.2 (0.7) 
3 9.4 (3.8) 17.8 (4.0) | 8.6(2.9) | 7.8 (1.9) 1.2 (0.7) 1.4 (1.0) 
4 18.2 (13.5) |9.2 (3.7) |160(3.8) 14.6 8.4) -1.0 (0.7) | —0.6 (1.7) 
5 13.8 (8.2) |7.8(4.2) | 9.8(2.9) | 9.0 (1.3) 1.8 (1.5) 1.8 (1.0) 
6 5.2 (4.2) |2.6(1.7) | 9.2.7) | 84(1.9) 2.0 (1.5) 2.0 (0.9) 
7 18.0 (17.8) |8.0(4.3) | 9.8 (3.5) | 8.0 (1.7) 0.8 (1.6) 2.0 (0.6) 
8 10.607.) 44a N |942 | 9022 1.2 (1.2) 1.4 (1.0) 
9 2.4(1.2) 2.2 (1.6) | 6.8 (0.7) | 8.0(1.4) 2.4 (0.8) 1.8 (1.5) 
10 5.2 (3.5) 12.8 (1.8) | 8.6(2.6) | 9.6 (2.5) 0.8 (1.2) 1.0 (0.6) 
11 30.4 (42.6) 16.8 (1.6) 116.6 (4.1) 1488.5) -140.7 |-1.4(17) 
12 3.0 (0.6) |28 (1.0) | 7.6(1.6) — 8.2(1.5) 2.2 (1.2) 1.8 (1.0) 
13 18.4 (11.8) |8.4(1.4) | 7.00.6 | 7.6 (1.4 2.4 (0.8) 2.2 (0.7) 
14 39.4 (42.9) |9.6(2.2) | 13.4(3.6) 12.8 (3.0) 0.6 (1.7) 0.8 (1.7) 
15 11.8 (7.3) |7.6(4.1) | 6.6(0.8) | 7.0 (0.6) 2.6 (0.8) 2.6 (0.8) 


Aver | 11.5(11.7) |6.2(2.8) | 9.7 (2.6) 9.4 (2.1) 1.3 (1.2) 1.4 (1.1) 


3.1.3 Satisfaction 


ARS scores averaged 1.6 for both LPWs. Both LPWs were analyzed close to a moderately 
acceptable level. (Table 4). As a result of SUS analysis, LPW1 scored averaged 77.6 
points (Black line), which was measured as grade B/’Good’ usability. LPW2 scored 
averaged 86.4 points (Red line), which was measured as grade A/’Best imaginable’ 
usability (Fig. 4). 


3.2 S1, S3 Tasks Performed by the Caregivers 
3.2.1 Effectiveness 


All caregivers performed successfully in all scenarios at once. Thus, LPW1 and LPW2 
showed 100% effectiveness. 
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Fig. 4. Result of S2 SUS (Color figure online) 


Table 4. Result of S2 ARS 


ARS (Aver. / SD) 


Task LPW1 LPW2 Task LPW1 LPW2 
1 2.6 (0.8) 2.6 (0.8) 9 2.2 (1.2) 2.0 (1.5) 
2 2.2 (0.7) 2.4 (0.8) 10 1.0 (1.3) 1.0 (0.6) 
3 2.0 (0.9) 2.4 (0.8) 11 -1.2 (1.7) —1.2 (1.7) 
4 —0.8(1.6) —0.6 (1.7) 12 2.4 (0.8) 1.8 (1.0) 
5 2.2 (0.7) 1.2 (1.8) 13 2.4 (0.8) 2.2 (0.7) 
6 2.4 (0.8) 2.0 (0.9) 14 1.2 (1.0) 1.6 (0.8) 
7 0.8 (1.7) 2.2 (0.7) 15 2.8 (0.4) 2.6 (0.8) 
8 1.8 (1.2) 2.0 (0.9) Aver 1.6 (1.0) 1.6(1.0) 


3.2.2 Efficiency 


TPT were averaged 6.7 s (LPW 1) and averaged 7.9 s (LPW2). RPE scores averaged 7.3 
points for LPW 1 and 6.7 points for LPW2. Both LPWs were usable with extremely light 
effort. DRS scores averaged 2.9 for both LPWs. Both LPWs were analyzed close to a 
very easy level. 


3.2.3 Satisfaction 


ARS scores averaged 3 points for LPW1 and 2.9 points for LPW2. Both LPWs were 
analyzed close to a very acceptable level. 


4 Conclusion 


Wheelchair users face a number of obstacles when traveling in a community environ- 
ment. Through this comparative usability testing, the usability of two LPWs developed 
in Korea was examined from the perspective of users and caregivers. From the user’s 
point of view, the usability was investigated by driving in a driving environment that can 
be experienced when moving to the community using a LPW, and by setting the battery 
and cushion of the LPW from the caregiver’s point of view. 

There was a difference between the two LPWs in terms of effectiveness and satis- 
faction in the participant’s driving, and there was no significant difference in efficiency. 
There was no significant difference between the two LPWs in the performance of the 
caregivers. 
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As a result of the usability testing, the usability of the heavier LPW2 was highly 
evaluated. This can be interpreted that users feel a sense of stability in the LPW with 
a certain amount of weight. In future studies, it is considered necessary to study the 
appropriate weight of LPW that users can feel stable. 

This study has a limitation in that it did not include powered wheelchair users with 
various types of disabilities by limiting the subjects to people with physical disabilities. 
It is believed that future research needs to evaluate usability considering various types 
of disabilities. It is expected that this study will be used as basic data in future research 
on LPWs. 
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Abstract. Recently, care robots are being developed that incorporate robotics into 
assistive products that focus on daily care for the physically disabled or elderly 
with reduced physical function. However, although care robots can reduce the 
physical burden of human intervention, they can also be dangerous depending 
on their situational awareness. This study describes a standardization that defines 
safety requirements for care robots and includes an evaluation method to test 
their safety based on these requirements. As an example of the application of this 
standard, a standard for the safety and performance method of a meal assistant 
robot is shown. This standardization study is expected to contribute to the spread 
of care robots in the future. 


Keywords: Care robot - Standard - Safety - Performance - Requirement 


1 Introduction 


The classification and terminology of assistive products are addressed explicitly in the 
ISO 9999 standard [1]. The standard defines an assistive product as “optimizing a per- 
son’s functioning and reducing disability.” In other words, an assistive product can 
optimize a person’s functioning or reduce disability. With the recent development of 
robotics, many assistive devices are incorporating robotics to improve their functionality. 
The terms robot and robotics are well-defined in ISO 8373:2021 [2]. 

A robot is defined as a “programmed actuated mechanism with a degree of autonomy 
to perform locomotion, manipulation or positioning,” and robotic technology is defined 
as “practical application knowledge commonly used in the design of robots or their 
control systems, especially to raise their degree of autonomy” [2]. Since IEC/TR 60601- 
4-1 defines amedical robot as a “robot intended to be used as medical electrical equipment 
or system,” an assistive robot can be defined as a “robot intended to be used as an assistive 
product” [3]. Similarly, a care robot is a “robot intended to be used for caring persons 
with disability or elderly persons.” 
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These care robots are intended to support the care, nursing, and physical activities of 
the care recipient, easing the burden of caregiving and improving the quality of life of the 
elderly or disabled and their families. Care robot products are developed around basic 
activities such as Feeding, Personal toileting, Hoist, and Posture-changeable. However, 
there are still no clear definitions, item classifications, and safety standards for care 
robots, so their dissemination has many difficulties. 

In this study, a care robot is defined as a “robot that assists the daily life of the disabled 
or the elderly who have difficulties in daily life.” However, although care robots can be 
physically burdensome by reducing human intervention, they can also be dangerous 
depending on their ability to recognize situations. Therefore, we propose a standard that 
defines safety requirements specific to care robots and includes an evaluation method 
to test safety based on these requirements. As an example of applying this standard, we 
show a standard for Feeding robots’ safety and performance methods. 

As for standards related to care robots, ISO 21856, the common standard for assistive 
devices, was recently published [4]. It consists of 29 clauses, with general requirements 
in clause four and individual requirements in clauses 5 through 26. The international 
standard for medical rehabilitation robots is IEC 80601-2-78 [5]. This standard adds 
mechanical hazards defined by robots to the mechanical hazards in IEC 60601-1, a 
common standard for medical devices. It is necessary to refer to it in the care robot 
standard because it considers the characteristics of robots. IEC 60601-1, a common 
safety standard for all medical devices, states that the primary user may be a patient rather 
than a clinical professional. Therefore, for medical robots with a degree of autonomy 
(DOA), the user’s situational awareness can be an essential safety factor [6]. Since the 
primary users of care robots are non-expert care receivers, and caregivers, the risk of 
losing situational awareness should also be considered. Methods for evaluating this risk 
include usability engineering in the field of medical devices [7] and usability evaluation 
in the field of assistive devices. 

Medical devices, assistive devices, and children’s products that have been used for a 
long time have well-documented safety requirements. However, the field of care robots, 
which has a short history, still lacks safety standards. This study proposes standards for 
safety and verification methods suitable for care robots by referring to existing standards. 


2 Safety Standard for Care Robots 


2.1 Definition of Care Robot 


Care or care service is defined as “the act of providing physical Care or care service as 
“the act of providing physical or mental assistance to the elderly or disabled who have 
difficulty maintaining daily activities on their own.” 

Thus, a care robot can be defined as “‘a robot that assists the daily life of the disabled 
or the elderly who have difficulties in daily life.” 


2.2 Types of Care Robots 


Care robots include Feeding robots, Hoist robots, Personal toileting robots, posture- 
changeable robots, and Rollator robots to reduce the burden on caregivers and improve 
the quality of life of care recipients (Fig. 1) [9]. 
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Fig. 1. Types of care robots [9] 


2.3 Structure of Care Robot Safety Standard 


The safety standards for care robots are divided into common safety standards, individual 
safety standards, care data standards, and care robot degrees of autonomy (Fig. 2). 
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Fig. 2. Hierarchy of care robot standards 


The common safety standard for care robots covers general safety requirements that 
should be applied to care robots. In contrast, the individual care robot standard covers 
requirements that are difficult to apply in general in consideration of the characteristics 
of individual care robots and must be considered for each item. Therefore, the Care Data 
Standard and the Care Robot Degree of Autonomy Standard are different standards that 
can be referenced for care robots. 


2.4 General Requirements 


For risk analysis and management of care robots, the procedures of ISO 14971 or ISO 
12100 are applied [10, 11]. In addition, for care robots classified as medical devices, IEC 
62366-1 is used to identify hazardous situations [7], and for care robots with embedded 
programs, IEC 62304 is used [12]. In particular, for care robots with autonomy, the loss 
of the user’s situational awareness is considered a hazard [13]. 
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Multifunctional care robots with two or more functions must meet the safety and 
performance requirements in individual standards according to the primary function pro- 
posed by the manufacturer. Functions other than the primary function shall be classified 
as minor and meet common safety requirements. 

If the care robot supports wired and wireless communication (Wi-Fi, Bluetooth, USB, 
RS-232, LAN, etc.) capabilities, it must be designed for availability, confidentiality, and 
integrity within the risk management process. 


2.5 Safety Requirements for Care Robots 


Safety requirements are categorized into electrical safety, mechanical safety, cleaning 
and disinfection, environmental factors, and hazardous materials. 

Electrical safety is required to comply with national safety certification requirements. 
If there are no national safety certification requirements, at least meet the requirements 
for leakage current, withstand voltage, and over temperature. In addition, built-in batter- 
ies, emergency stop, protective stop, and electronics compatibility are required (Table 1). 
Finally, mechanical safety requirements are provided for the structure, pinching, squeez- 
ing, and restraint to ensure that the user of the care robot is not at risk of mechanical 
harm. 

Requirements for hazardous substances are provided for parts of the robot that come 
into contact with the human body. The allowable values of hazardous substances are 
based on the Korean certification (KC) standard for children’s products [8]. 

To prevent harm from noise, the standard requires a maximum noise level of no more 
than 50 dB in sleeping environments, 65 dB indoors, and 85 dB outdoors. In addition, 
the standard requires classification for water resistance according to IEC 60529 [13]. 


2.6 Example of Feeding Robot 


This section introduces the individual requirements of a feeding robot. The shape of a 
feeding robot is shown in Fig. 3. The safety of the feeding robot is further tested for 
impact energy, static load, and repetitive durability (Table 2). The performance of the 
feeding robot is further evaluated through pose accuracy (ISO 9283) [14], continuous 
use time, and feeding assistance success rate (Table 3). 


Fig. 3. Feeding Robot 
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Table 1. Electrical Safety Requirements for care robots 


Item 


Built-in battery 


Requirements 


- Care bots that use batteries must have a battery status indicator. In 

addition, all circuits connected to the battery must be protected against 

overcurrent for user safety 

- The internal battery must meet the following requirements 

a) Batteries must not ignite, explode, or leak while using the robot 

b) All batteries must be completely sealed. If there are exposed terminals, 
they must be of opposite polarity and spaced at least 6 mm apart 

c) Battery terminals and connections must be sealed to prevent short 
circuits 


d) Lithium-ion batteries must meet IEC 62133-2 [15] 


Emergency stop 


- The care robot shall be equipped with one or more emergency stop 

devices, and the emergency stop shall meet the following requirements 

a) The emergency stop device must reduce the risk to an acceptable level 

b) The E-Stop device must be easily accessible to the user 

c) The E-STOP device must be colored red 

d) The E-STOP device shall be marked on or near its surface with the 
symbol IEC 60417-5638 or the word STOP [16] 

e) Once activated, the emergency stop device shall remain stationary until a 
user action is taken to restart it 


Protection stop 


- The care robot shall have a protective stop function if required by risk 

management. The protective stop function shall comply with the following 

requirements 

a) The protective stop must reduce the risk to an acceptable level 

b) A program embedded in the care robot must automatically restart the 
protective stop. However, the user must initiate the restart manually if 
the automatic restart poses an unacceptable risk 
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Table 2. Additional Safety Requirements for Feeding robot 


Item Requirements 

Impact energy - The permissible impact energy for body contact of the robot arm shall 
follow ISO/TS 15066 

Static load - Place the feeding robot in the most unfavorable position for installation 


using the manufacturer’s suggested method and hold it for 60 s with 1.5 
times the manufacturer’s suggested payload 
- The feeding robot must not be damaged or destroyed after the test 


Repetitive durability | - 10 000 repetitions of the manufacturer’s route at the rated speed with 
the manufacturer’s maximum payload. An external power source other 
than battery power may be used for the continuity of the test 

- The feeding robot must maintain the manufacturer’s performance after 
the test 


Table 3. Performance Requirements for Feeding robot 


Item Requirements 


pose accuracy - The manufacturer provides the commanded pose (start pose 
and end pose) of the feeding robot 

- Repeat the command pose ten times with the maximum 
carrying load and rated speed suggested by the manufacturer, 
and measure the reaching pose of the meal assistant robot. 
Currently, the pose accuracy is expressed as the deviation 
from the average of the command pose and the reaching pose 
according to ISO 9283 

- The pose accuracy is recorded in the test report 


continuous use time - Test the driving pattern according to the manufacturer’s 
expected usage 

- The battery shall be fully charged before the test 

- Apply the maximum payload specified by the manufacturer 
and measure the continuous use time with a second-hand 
clock 


feeding assistance success rate | - The manufacturer shall provide the type, shape, nature, and 
size of the food to be tested, as well as the assisted feeding 
maneuvers, reaching poses, and routes for scooping and 
transporting the food to the destination 

- When the tester uses controls such as switches or buttons to 
specify a food item, scoops up the specified food item, and 
transports it to the destination, measure the following 
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3 Conclusion 


Recently, various research projects on care robots have been started and are in progress. 
However, due to the lack of relevant standards, there may be problems with the steps for 
licensing or certification for future productization. Therefore, we developed a standard 
for safety requirements for care robots in this study. 

The requirements for electrical safety and mechanical safety, which are safety 
requirements for medical devices, were investigated and applied [6]. The mechanical 
risk factors defined by robots were additionally reflected by referring to ISO 13482 [16]. 
In addition, since care robots use robotics and the primary users are non-experts, the 
risk of situational awareness loss was considered by referring to IEC 80601-2-78, an 
international standard for rehabilitation robots [5], and cybersecurity was also added. Per- 
formance issues will be addressed in individual standards for care robots to be developed 
in the future. 

In a situation where standardization of safety evaluation of care robots is actively 
needed, it is expected that securing product safety and consumer trust through normal 
development will be very important for the successful settlement and expansion of the 
care robot industry. Furthermore, the developed standards can be used for licensing and 
establishing policies for care robots, and manufacturers will be able to ensure safety in 
the development stage of care robots. 

The addition of robot and robot technology definitions, automatic toileting systems, 
power assist units for assistive products for walking, and feeding robots to ISO 9999 
has paved the way for developing and institutionalizing international standards for care 
robots [1]. Therefore, it can be promoted as an international standard based on the current 
standards. In addition, we will also develop standards for individual care robot items 
so that care robots currently under development can be released to the market with 
guaranteed safety and performance. 
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Abstract. Assistive technology devices (ATD) and services (ATS) are generally 
applied to assist the people with disabilities and the elderly who have limited 
independent performance in daily life. The Korean government is also providing 
support through various projects, but the evidence for its effectiveness is lacking, 
and its application is also limited. Therefore, it is necessary to objectively measure 
the value of ATD and present evidence. This study analyzed studies that measured 
the economic and non-economic values of ATD and ATS through a systematic lit- 
erature review, and looked at the overall economic analysis process. The database 
was compiled using KCI, NDSL, Web of Science and Scopus. Nineteen studies 
were selected according to a set procedure for literature screening, and the quality 
level of studies was analyzed through the CHEERS checklist 2022. Most studies 
have analyzed hearing aids, and most studies have used cost-effectiveness tech- 
niques. Measurement instruments were most commonly used to measure health 
and quality of life, accounting for approximately 60% of all. The quality level 
of the studies was calculated as ‘Good’ and ‘Average’ grades. In the future, it is 
expected that a high-quality economic analysis study will be conducted to measure 
the economic and non-economic values of various ATD and ATS in Korea. 


Keywords: Assistive Technology - Economic Analysis - Literature Review 


1 Introduction 


The World Health Organization emphasized that efforts to remove environmental and 
social barriers are needed to overcome the difficulties experienced by people with dis- 
abilities, and it has been confirmed that the people with disabilities rely on other people 
or assistive technology devices when necessary to carry out their daily activities [1]. 
The Korean government supports assistive technology device for the people with 
disabilities and the elderly through 9 projects in 4 departments (Ministry of Health and 
Welfare, Ministry of Employment and Labor, Ministry of Science and ICT, Ministry of 
Education) and | department (Ministry of Patriots and Veterans Affairs). Researchers 
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reported that the use of assistive technology improves independence and quality of life, 
affects family and social burdens, and reduces the help of others [2-5]. This means that 
it is important to provide appropriate and effective assistive technology devices accord- 
ing to the needs and abilities of the subjects, and in-depth evaluation by occupational 
therapists, regular inspection of assistive technology devices, and cooperation with other 
experts are required [6, 7]. 

In Korea, the health insurance budget was gradually estimated from 2017 to 2022 
to alleviate the burden of the vulnerable, and in particular, health care plans to allevi- 
ate economic difficulties and cost burdens were emphasized to ensure people-centered 
universal health [8—10]. Accordingly, plans and measures to efficiently use the limited 
budget of the state are required [11]. Kong and Jung [12], cost-effectiveness, cost-utility, 
and cost-benefit analysis are common techniques for confirming economic feasibility. 
However, related research in Korea is very scarce. According to WHO [2], through the 
support of appropriate assistive technology devices, the state not only reduces medical 
costs and additional welfare budgets due to repeated hospitalization of users such as 
the disabled and the elderly, but also provides socio-economic value by enabling users’ 
economic activities reported to be able to create. 

Therefore, this study analyzes the economic feasibility of providing assistive technol- 
ogy devices or services, systematically reviews studies that derive economic and non- 
economic values, and analyzes economic analysis techniques and measurement tools 
used in the economic analysis. Through this, plans to efficiently use the government’s 
limited budget in the field of rehabilitation in Korea will be presented. 


2 Method 


2.1 Database and Search Term 


The databases for literature search for this study are the National Digital Science Library 
(NDSL), Korea Citation Index (KCI), Scopus, and Web of science. Articles included 
domestic and foreign academic journals published from February 2018 to February 2023 
based on the publication date. 

Search terms are (“Economic analysis” OR “economic evaluation” OR “Cost- 
benefit” OR “Cost-effectiveness” OR “Cost-efficiency”) AND (“Assistive device” OR 
“Assistance tool” OR “Assistive Technology device” OR “Assistive Technology service” 
OR “Assistive Technology”) was searched, and it was translated into Korean and used 
in the KCI and NDSL databases. Papers retrieved from each database were integrated 
through Endnote 20, duplicated articles were removed, and then selected based on the 
following inclusion and exclusion criteria. 
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2.2 Selection Criteria 


Each article was selected if it satisfies the following criteria. 


(1) Articles published in academic journals 

(2) The article analysing the economic value of ATD or ATS 

(3) The article analysing the non-economic value of ATD or ATS 
(4) Thesis written in Korean or English 


And, if each article satisfies the following criteria, it was excluded. 


(1) The article analysing only the effectiveness of ATD or ATS 
(2) Review or meta-analysis research, degree research, protocol research thesis 
(3) Articles for which full text cannot be obtained 


Since this study only considered studies analyzing the economic feasibility of assis- 
tive technology devices and services and studies reviewed through peer review, system- 
atic reviews, meta-analyses, and degree studies were excluded. In addition, articles that 
could not be read or whose economic feasibility was not analyzed were excluded. The 
literature was independently reviewed by two researchers and finally selected through 
discussion. Finally, 19 articles were selected (Fig. 1). 


2.3 Quality Level of Research 


The quality level of the final selected articles was evaluated using the CHEERS (Con- 
solidated Health Economic Evaluation Reporting Standards) checklist. The checklist 
was announced in 2013, the qualitative level of the economic evaluation was evaluated 
by Drummond et al. [13], after responding with ‘Yes’, ‘No’ or ‘Not Applicable’, if the 
item meets the guidelines, 1 point (yes), and half of it, 0.5 points (partial), if not sat- 
isfied at all, it can be calculated as 0 points (no). As the checklist was revised in 2022 
through an expert Delphi survey from 2020, so the 2022 version of the revised CHEERS 
checklist was used in this study. The CHEERS checklist 2022 has added items related 
to patient or service recipient, general public, community or stakeholder engagement 
and involvement, and terminology has been expanded for broader applicability. And 
modifications have been made to make the analysis more transparent and to help readers 
decision-making, including items such as whether the economic analysis was planned 
and whether the model used could be used by anyone [14]. 


2.4 Analysis 


In this study, the frequency of Type of Assistive Technology Device (ATD) or Service 
(ATS), Type of economic analysis methods, and Measurement tools used in the article 
was analyzed, and the Results of the Quality level of research were analyzed. 
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3 Result 


3.1 Analyzed Assistive Technology Device (ATD) or Service (ATS) 


Among the types of assistive technology devices and services analyzed in the final 
selected literature, hearing aids showed the highest frequency, and the study was con- 
ducted targeting the people with hearing disabilities due to disease and the elderly 
population. Next, there were many analyzes on assistive technology, hearing service, 
and manual wheelchair. In addition, the economics of vehicle modification, prosthesis, 
rollator, and upper limb robot was analyzed in one article each (Table 1). 
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Table 1. Type of Assistive Technology Device or Service. 


Type of ATD or ATS N (%) 
Hearing aids 9 (45.0) 
Overall Assistive Technology 3 (15.0) 
Hearing service 2 (10.0) 
Manual Wheelchair 2 (10.0) 
Modified Vehicle Service 1 (5.0) 
Prostheses 1 (5.0) 
Rollator 1 (5.0) 
Upper limb robotic 1 (5.0) 
Total 20 (100.0) 


3.2 Used Economic Analysis Methods 


The techniques used in the analysis were cost-effectiveness, cost-benefit, and cost-utility 
analysis, and cost-effectiveness techniques were used most frequently. Among the eco- 
nomic analysis methods mentioned above, the cost-effectiveness method was used the 
most, and it was also used in various assistive technology device and service studies, but 
the high proportion of the papers on economic analysis on hearing aids had an impact. 
Additionally, there was one article each on the social value of assistive technology and 
telecare (ATT), and vehicle modification (Table 2). 


Table 2. Type of Economic Analysis Method 


Type of economic analysis N (%) 
Cost-effectiveness (CEA) 10 (50.0) 
Cost-benefit (CBA) 6 (30.0) 
Cost-Utility (CUA) 2 (10.0) 
ETC (social value) 2 (10.0) 
Total 20 (100.0) 


3.3 Measurement Tools by Cost and Outcome 


In the analyzed studies, costs were generally determined through unit costs, based 
on national surveys or published data. Direct costs typically included initial consul- 
tation, intervention costs such as assistive technology devices or services, evaluation 
and training, and, if necessary, surgery costs. Indirect costs included transportation and 
opportunity costs for individuals and caregivers, and non-working hours. 
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Outcome tools measured as a result of economic analysis were analyzed. Of the 11 
tools, the tools to evaluate health and quality of life accounted for a high proportion of 5, 
and among them, quality adjusted life year (QALY) was used the most. QALY is a well- 
known measure of the degree to which a treatment or system improves quality of life 
while prolonging lifespan, and it has been pointed out that it is a measure tailored to the 
individual rather than society. Next, EQ-5D and HUI3 were used as questionnaire tools 
to measure health utility. The EQ-5D responds to mobility, self-care, usual activities, 
pain/discomfort, and anxiety/depression. Similarly, HUI3 responds to the state of eight 
attributes, such as vision, hearing, and ambulation, etc. And according to the purpose 
of the study, tools to evaluate elderly people with dementia and tools to evaluate people 
with disabilities who are working were used. 

In addition, five articles analyzed complications prevented through assistive tech- 
nology devices and services, cost reduction, and durability of devices themselves as 
outcomes, and two articles that studied social values measured non-economic values 
through questionnaires and CVM (Table 3). 


Table 3. Outcome Measurement tool of included studies 


Measurement tool N (%) 
Quality Adjusted Life Year (QALY) 8 (27.6) 
Health Utilities Index Mark 3 scale (HUI-3) 4 (13.9) 
Health Related Quality of life (EQ-5D) 4 (13.9) 
Questionnaire 2 (6.9) 
Quality Adjusted Work Life Year (QAWLY) 1 (3.4) 
Contingent Valuation Method (CVM) 1 38.4) 
Geriatric Depression Scale (GDS) 1 38.4) 
Clinical Dementia Rating (CDR) 1 (3.4) 
Work Limitations Questionnaire (WLQ-25) 1 38.4) 
Workplace Activity Limitation Scale (WALS) 1 (3.4) 
ETC (prevention, saving money, durability) 5 (17.3) 
Total 29 (100.0) 


3.4 Results of the Quality Level of Research 


The total score range of CHEERS 2013 checklist ranges from 0 to 24, and higher scores 
mean excellent quality levels, and can be divided into excellent (24 points), good (17.5 
to 23 points), average (11.5 to 17 points), and poor (11 points or less) [13]. However, 
since the revised CHEERS 2022 checklist has not yet been presented and consists of 
28 items, this study classified it into excellent (28 points), good (20.5-—27.5 points), 
average (13.5—20 points) and poor (13 points or less) based on previous studies. As a 
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result of the analysis, the total score ranged from 17 to 27 points, and one of the four 
studies conducted according to CHEERS checklist scored the highest with 27 points. Two 
articles that studied the cost-benefit of lightweight manual wheelchairs were excluded 
from the analysis because only limited information was provided. Therefore, 17 articles 
were evaluated in this study (Table 4). 


Table 4. Level of Quality of included studies 


Grade Range of score Frequency (%) 
Excellent Score 28 point 0 (0.0) 

Good Score 20.5 — 27.5 point 12 (70.6) 
Average Score 13.5 — 20.0 point 5 (29.4) 

Poor Score under 13 points 0 (0.0) 

Total 17 (100.0) 


4 Conclusion 


The application of assistive technology devices and services has a great impact on the 
lives of people with disabilities, improves their lives, and enables new attempts, so they 
must be attempted. The Korean government is also making great efforts for this, but 
so far, limited application is being made. Therefore, this study attempted to provide 
basic data for improving the existing system by systematically reviewing studies that 
analyzed the economic and non-economic values of assistive technology devices and 
services. As a result of analyzing previous studies, both economic and non-economic 
values of assistive technology devices and services for the people with disabilities were 
measured, and the economic value was mainly calculated through a unit cost based 
on national survey and published data, and the non-economic value was calculated 
through QALY, questionnaires and conditional value evaluation (willing to pay). Also, 
Outcome measurement tools were mainly related to health and quality of life. Through 
this, the government will provide efficient support without wasting budget, and people 
with disabilities can expect to improve their quality of life by reducing the burden in 
purchasing and using necessary assistive devices and services. Lastly, In Korea, there 
are few studies that have confirmed the effectiveness, benefits, and utilities of assistive 
technology devices and services on the health and life of people with disabilities who use 
and receive them. Accordingly, it is expected that researches that measure the economic 
and non-economic values of various ATDs and ATSs in Korea according to an objective 
checklist will be actively conducted in the future. 
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Abstract. Edge computing, a distributed computing architecture 
within the knowledge-defined network (KDN), faces challenges due to 
the significant disparities and data heterogeneity among its nodes, hin- 
dering their interaction. Ontology, a solution within the Semantic Web, 
is well-suited for addressing data heterogeneity and matching ontolo- 
gies effectively. However, ontology matching presents difficulties due to 
non-linear mathematical issues. To overcome these challenges, the gen- 
erative adversarial network (GAN), an unsupervised learning method, 
has emerged as a promising tool. GAN consists of two models with dis- 
tinct objectives trained against eachother to achieve optimal outcomes. 
This paper introduces SA-GAN, an algorithm that combines GAN with 
simulation-based annealing to enhance its effectiveness. SA-GAN utilizes 
a stagnation counter to expedite the convergence speed of GAN. Through 
experiments conducted on a renowned ontology benchmark, the paper 
demonstrates that SA-GAN, along with other ontology matching algo- 
rithms, can identify the best alignments. Consequently, SA-GAN facili- 
tates the construction of bridges in edge computing, improving its overall 
effectiveness. 


Keywords: Edge Computing - Ontology Matching - Simulation-Based 
Annealing Generative Adversarial Network (SA-GAN) 


1 Introduction 


The new paradigm of knowledge-defined networking (KDN) combines artificial 
intelligence (AI), software-defined network analytics (NA), and software-defined 
networking (SDN). Within KDN, edge computing empowers edge nodes with 
storage and processing capabilities to perform data forwarding, recognition, and 
other services. However, these edge nodes often have different data representa- 
tions, leading to data heterogeneity. Ontology, at the very exact moment, acts as 
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a model of reference for information transmission, allowing for the exact correct 
semantic standardized meaning to date [10]. To address this, edge node ontology 
is used to represent concepts and relationships in specific domains. Ontology, a 
fundamental technique in the Semantic Web, is widely employed to tackle data 
heterogeneity by effectively modeling knowledge and formalizing relationships 
and concepts. The System Upgrade Process in Fig. 1 is an example of how ontol- 
ogy is applied and mapped to interconnected concepts. Ontology helps identify 
similar domains, definitions, terms, and their connections, creating a graphical 
map of entities and their associations, including links to other corporate domains. 
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Fig. 1. An Ontology about System Upgrade 
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Ontology construction is subjective, resulting in heterogeneity in ontologies 
due to different techniques used for representing ideas and relationships [8]. In 
an information-rich era, multiple sources produce similar content, making it 
impossible for systems to have identical ontologies. Ontology matching provides 
a direct approach to leverage ontologies by identifying heterogeneous entities 
across multiple ontologies. When given two distinct ontologies with discrete items 
like attributes and classes, ontology matching involves determining relationships 
such as equality and formation among these entities [11]. 

Ontology meta-matching is a complex problem that involves determining 
suitable weights for combining multiple similarity measures, ensuring high- 
quality alignment. Unsupervised learning is well-suited for large-scale ontology 
meta-matching, as it doesn’t rely heavily on manual labeling [9]. Generative 
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adversarial networks (GANs) are popular in unsupervised learning. They consist 
of a generator and a discriminator. The generator produces simulated samples, 
while the discriminator provides feedback on their authenticity. Through itera- 
tions, the generator improves until the discriminator can’t distinguish between 
real and generated samples. GANs have found applications in various fields, 
including computer vision, and have inspired innovative approaches in ontology 
matching. 

Rest of the paper is organized in the following order: Sect.2 gives an out- 
line of the state of the art literature review where we outline the work done. 
Section 3 presents the model of adversarial learning based ontology alignment. 
While Sect.4 gives and outline for the experimental analysis we have done. 
Finally the paper gives a comparison to Instance Matching using Knowledge 
Graph in Sect. 5 and concludes the work in Sect. 6. 


2 Literature Survey 


Ontology matching bridges the semantic gap between different domain repre- 
sentations but can be time-consuming. The use of diverse modeling approaches 
is necessary due to the multiple perspectives in perceiving things. Integrating 
domains often involves representing combined knowledge using ontologies and 
employing ontology matching approaches to align them. Our literature review 
focused on recent publications in the past three years and revealed a widespread 
interest in ontology matching. Our objective was to classify and identify research 
trends in ontology matching while developing a reference framework for integrat- 
ing and categorizing these materials. 

Ontology matching bridges the gap between domain representations. Inte- 
grating diverse perspectives requires different modeling approaches using ontolo- 
gies. Our literature review identified trends and developed a framework. We 
examined a meta-strategy-based KG matching approach that aligns schema 
and instance-level entities. Experimental results favor algorithm-based meta- 
heuristic approaches over CNEA-based KG matching [11]. 

We introduce a heuristic evaluation measure and an optimization model 
for ontology matching. Our approach utilizes an Adaptive Evolutionary Algo- 
rithm (AEA) and outperforms reference alignment-based measures. It avoids 
local optima and excels in determining high-level alignments compared to exist- 
ing systems. Our AEA-based ontology meta-matching system generates superior 
alignments independently [4]. 

Early work on graph alignment employs a graph embedding algorithm and 
an absolute orientation rotation method. Studies have shown that this method 
is effective for aligning structurally similar ontologies and is more robust against 
alignment noise when dealing with graphs of different sizes and architectures. 
Future research aims to explore various embedding methodologies and integrate 
the method with additional features, such as text-based information, in a hybrid 
matching system [1]. 
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Normalization of Word and representation learning are employed to enhance 
the optimization of ontology alignment approaches using feature-based method- 
ologies. Word normalization helps capture various domain terms, while represen- 
tation learning discovers semantic and conceptual information, expanding align- 
ment possibilities. Entity embedding learning within the representation learning 
approach is investigated [2]. 

Ontology heterogeneity in Knowledge Delivery Networks (KDNs) hampers 
collaboration among edge nodes. Ontology matching tackles this challenge by 
assessing similarity. We propose an adversarial learning model based on simu- 
lated annealing for ontology meta-matching. It optimizes weights and thresholds 
using a Generator-Discriminator relationship, outperforming previous systems 
and achieving effective ontology meta-matching [3]. 

We propose a mapping procedure to generate a semantic-compatible ontology 
for Digital Dentistry (DD) using a base ontology and three reference ontologies. 
Additionally, we suggest a deep learning-based method to identify key factors 
contributing to depression in DD. Due to limited data availability caused by the 
COVID-19 pandemic, further experimentation is needed to validate the proposed 
methods [5]. 

DeepOM is a deep learning-based ontology matching system that handles 
large ontologies without partitioning. It generates concept embeddings using a 
reference ontology and an auto-encoder, resulting in accurate and compact rep- 
resentations. DeepOM outperforms other systems in matching large-scale ontolo- 
gies. The use of an auto-encoder for concept embeddings proves effective, with 
all DeepOM parameters contributing to improved matching [6]. 

This study proposes an approach for aligning IIloT ontologies using NLP. 
It learns vector embeddings for items and relations using ontology metadata 
and structure. Experimental results consistently outperform the baseline model 
BERT INT in HR and MRR scores by 1.2—-2.7%. Limitations include synthesizing 
the ontology dataset due to unavailability of real-world data. However, the model 
benefits from learning language embeddings, facilitating identification of nodes 
with comparable semantic meaning. The structural encoder accurately aligns 
nodes, eliminating biases and imposing context [7]. 


3 Model of Adversarial Learning Based 


In this study, the meta-matching problem of ontology is treated as a constrained 
optimization problem. To solve it, popular heuristic algorithms like simulated 
annealing, genetic algorithm (GA), and technique are commonly used. These 
algorithms excel in handling complex optimization problems, especially with 
large datasets (e.g., SA). 

The choice of a specific heuristic algorithm depends on the ontology’s char- 
acteristics and the optimization problem’s nature. Selecting the right algorithm 
is vital for achieving effective and efficient meta-matching of ontologies (Fig. 2). 

The use of genetic algorithms (GA) to solve the meta-matching problem in 
ontology has two main limitations: slow convergence and premature convergence. 
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Fig. 2. GANs Framework 
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These problems can reduce the algorithm’s effectiveness and efficiency in finding 
optimal solutions. 

To address these challenges, we propose a new approach called SA-GAN. It 
combines simulated annealing with the adversarial training framework inspired 
by generative adversarial networks (GANs). We use the Metropolis criterion to 
generate optimal characteristics for solving the ontology meta-matching problem. 
In the context of GANs, our methodology has two components: the Generator 
and Discriminator models. 


3.1 Objective Function 


The rate of recall within the domain of ontology matching is computed as the 
ratio of accurate alignments chosen from the overall count of alignments gener- 
ated by the ontology matching system. It assesses the system’s capacity to accu- 
rately recognize and incorporate pertinent matches in its outcomes. The recall 
rate provides an important evaluation metric for assessing the completeness and 
comprehensiveness of an ontology matching system’s results. It indicates how 
well the system captures the true matches in the given ontologies and helps in 
evaluating its performance. The following is a definition of the formula: 


len(AU R’) 
iF an (1) 
len(A) 
The proportion of accurate alignments we select the overall number of total 
alignments chosen is called precision. It assesses the ontology matching system’s 
precision. The following is the formula: 


recall = 


/ 

rena ah ) 2) 

en(R’) 

We expect recall and accuracy to be equal (both ranges are between 0 and 
1), but this isn’t always the case, therefore we included f - measure which is a 
harmonic average to assess the match of ontology outcomes. It even emphasizes 
our system’s performance in a more intuitive manner. In addition, the formula 
can be changed. the outcome is nearer to recall when it is nearer to the 0, while 
its result is nearer to the precision the nearer it is to 1. (Xue & Wang, 2015a) 
The function f-measure is given below: 


precision = 
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recall x precision 


(3) 


In the SA-GAN framework for ontology meta-matching, the goal of the Gen- 
erator is to generate optimal similarity thresholds and weights that lead to the 
optimization of alignment quality and maximize the value of the f-measure. The 
f-measure is a metric that combines precision and recall to assess the overall 
effectiveness of the ontology matching system. 

The ultimate objective of the SA-GAN framework is to achieve a suitable 
combination of weights and similarity criteria that yield good performance in 
terms of the f-measure. The Generator plays a crucial role in this process, guided 
by the Discriminator’s feedback on the f-measure. The framework involves inter- 
active training of both the Generator and Discriminator parameters to optimize 
the meta-matching process. 

Figure 3 illustrates the overall framework, highlighting the interactions and 
iterative training between the Generator and Discriminator to achieve effective 
ontology meta-matching. 
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Fig. 3. SA-GAN Framework 


In ontology meta-matching, when the generator’s weights and threshold make 
the discriminator’s precision and recall both 1, regardless of how it trains, the 
f-measure is 1, the discriminator will end training and now the last threshold 
and weights will be displayed out (Fig. 5). 


3.2 Simulated Annealing Optimizer 


This paper explores using simulated annealing, an optimization algorithm, for 
parameter optimization in ontology meta-matching. Simulated annealing is effec- 
tive for numerical optimization problems with many local minima. The authors 
propose utilizing simulated annealing as a generator optimizer to speed up train- 
ing and improve results. They suggest adjusting the update probability of the 
algorithm to enhance exploration of the parameter space and achieve better 
optimization outcomes in ontology meta-matching. Here E is either equal to 0 
or less than 0, now updated solution is superior to the previous solution, and the 
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efficient solution is thus revised. Otherwise, based on the existing temperature 
condition, it will accept the new answer with a fixed likelihood. In the opti- 
mization procedure for the generator, the authors introduce Algorithm 1, which 
describes the calculation process using simulated annealing with a modification 
to address efficient solutions. The goal is to find the best optimal solution within 
the SA-GAN epoch. 


aadi AE <0 
T ) e 4E/t AE >0 


In this algorithm, the temperature adjustment is controlled using a 
geometric-based reduction. The temperature (T) is updated using the formula T 
= current temperature/m 8, where m is a rate parameter of cooling. This geo- 
metric reduction allows the annealing process to speed up, gradually reducing 
the temperature. 

Simulated annealing accepts worse options at high temperatures to explore 
the search space and potentially reach the global maximum. The authors intro- 
duce a stagnation counter in Algorithm 1 to measure solution efficiency. If the 
same result occurs multiple times, it indicates a lack of progress and termina- 
tion of the algorithm. This modified approach aims to enhance the generator’s 
effectiveness in finding optimal solutions for ontology meta-matching within each 
SA-GAN epoch. 


3.3 Using Gradient Descent and Discriminator 


Because discriminator must be educated in the fmeasure, it is thought of as a 
univariate function optimization issue. We use gradient descent, iterative opti- 
mization which is considered to be first order process for calculating differentiable 
functions’ local minima, to swiftly identify the ideal case. 

Here algorithm gradient descent follow the rule i.e., in case the univariate 
function f-measure(q) is identified and it also is distinguishable in the neighbor- 
hood of the point a, f-measure(a) decreases with speed at point a in the differ- 
ent direction of its gradient, rfmeasure(qa). It follows that, if b = a — yAf — 
measure(a), now considering yRp less then f —measure(a) > rf —measure(a). 
keeping this in notice, it should begin with an initial approximation of its local 
minimum 29 with f — measure(a), so we know here is a sequence of x in a way 
that z0, £1, @2,-.-;Un,Un41 so that Xn41 = Xn —nAf — measure(X,),n > 0. 
Therefore, f —measure(x9) > f—measure(2,) > f —measure(x2), if it proceed 
good, its sequence is converged obtaining desired local minimum finally. 

We varied parameters’ range in its discriminator in the experiment, creating a 
wave at 0.46 to 0.56 that we chose from a sensitivity test approximately. Because 
we aim to enhance both the precision rate and the recall rate during the training 
of parameter, here, we even need the end outcome to be optimized in the last 
f-measure calculation, which is 0.51. 
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3.4 Training Process of Model 


SA-GAN, a revolutionary adversarial learning model that offers a fresh app- 
roach to the problem of ontology meta-matching. The flowchart related to our 
process of matching is shown in Fig. 5 during the stage of pre-processing, it fin- 
ishes extracting alignments of ontology and calculating our matrix of similarity. 
Then we convert matrix of similarity in our data set which allows our model to 
be trained. The generator and discriminator models are then trained using an 
adversarial technique in the training step. Finally, to achieve the final ontology 
alignments, the weights and threshold produced during adversarial training were 
combined for similarity computation. 


4 Experimental Analysis 


In this work evaluated their ontology matching strategy using a benchmark 
track provided by the Ontology Alignment Evaluation Initiative (OAEI). They 
compared their generated alignments with the correct correspondences in the 
reference alignment to measure performance. This allowed for a comprehensive 
evaluation and comparison with other matching systems in the OAEI bench- 
mark. 

Table 1 gives a detailed overview regarding OAEI benchmark. Here Table 1’s 
first column shows the ID in our test cases, which corresponds to several sorts 
of ontologies in the experiment. We can discover that our system is sensitive to 
which features and our system is inconsiderate to which features to using various 
sorts of test sets, and we can make more precise adjustments to the system based 
on this information (Fig. 4). 

Here SA-GAN model is undergone training for maximum 50 epochs in the 
adversarial training step. Set Max Temperature to 200, Min Temperature to 
100, Maximum Stay Times to 30 and iterations to 99 for fast-SA optimizer. The 
weights and threshold are Set for the generator at random initial is set to 0.51 
and range is set to 0.46 to 0.55 for the discriminator, and epochs and learning 
rate for the algorithm gradient descent to 11, 0.01 correspondingly (Fig. 6). 


Table 1. OAEI’s Benchmark Description 


Cases ID | Introduction 
101-104 | Ontologies which are Consistent. 


201-208 | Ontologies not having similar linguistic and lexical features. 


221-247 | Ontologies with different structure features. 


248-266 | Ontologies not having similar structure, linguistic, lexical and features. 
301-304 | Ontologies of Real world 
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Cases ID 0.000 
P 
101-104 1.000 
201-208 0.890 
221-247 0.987 
248-266 0.780 
301-304 0.890 
Cases ID Edna 
101-104 1.000 
201-208 0.546 
221-247 0.884 
248-266 0.350 
301-304 0.462 
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Fig. 4. SA-GAN Flowchart 
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Fig. 6. OAEI’s Benchmark for experimental Analysis 
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4.1 Experimental Results 


In the experiment, the evaluation function’s parameter varies, requiring iden- 
tification of a suitable range for the discriminator’s training parameter. The 
objective is to control the parameter within a manageable range while improving 
both recall and precision in ontology matching results. Table 2 presents results 
achieved in various benchmarks with six groups in the range. Adjusting the range 
between 0.45 and 0.55 enhances precision and recall. 

Researchers compare their method’s benchmark results with participants 
from OAEI, including CODI, MapPSO edna, GeRMeSMB, TaxoMap, Falcon, 
and AROMA. Table 3 shows their method performs well in five types of OAEI 
benchmarks. However, GeRMeSMB’s results in 201-208 differ slightly from Fal- 
con’s in 301-304 due to the lack of entity context and structure consideration, 
along with imprecise semantic-based similarity metrics. 


5 Comparison to Instance Matching in Knowledge Graph 


Problem Focus: While the proposed framework focuses on ontology meta- 
matching, which involves aligning and optimizing different ontologies, instance 
matching in the knowledge graph aims to identify and match specific entities or 
instances across different knowledge graphs. 


Optimization Approach: The researchers in the proposed framework utilize 
heuristic algorithms such as simulated annealing and genetic algorithm to solve 
the complex optimization problem of ontology meta-matching. These algorithms 
iteratively adjust weights and similarity criteria to improve the alignment results. 
On the other hand, instance matching in the knowledge graph typically relies 
on graph-based algorithms and similarity measures specific to entity attributes 
and relationships. 


Evaluation Metric: The framework’s evaluation metric revolves around the 
f-measure, which considers both precision and recall. The objective is to achieve 
an f-measure of 1 by iteratively adjusting the weights and thresholds using the 
Generator and Discriminator. In contrast, instance matching in the knowledge 
graph often employs metrics like precision, recall, and Fl1-score at the instance 
or entity level. 


Training Process: The proposed framework’s training process involves inter- 
active training of the Generator and Discriminator parameters. The Generator 
aims to find optimal weights and similarity criteria, while the Discriminator pro- 
vides feedback by assessing the alignment results. Training continues until the 
Discriminator’s precision and recall reach 1, indicating a successful match. In 
instance matching, the training process focuses on learning and optimizing sim- 
ilarity measures and matching rules specific to the attributes and relationships 
of instances. 


Scope and Data Considerations: The proposed framework deals with ontol- 
ogy meta-matching and addresses the issue of semantic heterogeneity induced by 
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knowledge representation differences. It requires access to ontological metadata 
and structures to guide the alignment process. Instance matching in the knowl- 
edge graph, on the other hand, focuses on identifying correspondences between 
specific instances, relying on available attributes and relationships within the 
knowledge graphs. 


Applicability and Limitations: The proposed framework enhances ontology 
meta-matching, achieving a high f-measure through optimized alignment results. 
It has been evaluated on benchmark datasets, but its effectiveness may vary 
depending on ontology characteristics and available metadata. On the other 
hand, instance matching techniques in the knowledge graph target specific enti- 
ties within the graph, each with its own strengths and limitations based on the 
chosen approach. By including this detailed comparison section, readers gain 
insights into the distinct contributions and limitations of the proposed frame- 
work for ontology meta-matching and instance matching in the knowledge graph. 


6 Conclusion and Future Work 


Semantic heterogeneity in knowledge-defined networks arises from differences in 
knowledge representation, impacting collaboration among edge nodes. Ontology 
matching addresses this issue by determining weights and confidence levels for 
multiple similarity assessment methodologies. This study proposes a simulated 
annealing-based adversarial learning framework for ontology meta-matching. It 
optimizes a single-objective model by iterative tuning of parameters and achiev- 
ing improved matching outcomes compared to previous systems. 

Future work aims to enhance ontology alignments by considering entity struc- 
ture and improving confidence value accuracy. Addressing the slower convergence 
of the framework during training is also a focus, involving parameter adjustments 
and optimization for faster convergence. 
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